Wheldall blog what's wrong with What Works C/house

Moderators: Debbie Hepplewhite, maizie, Lesley Drake, Susan Godsland

yvonne meyer
Posts: 736
Joined: Tue Oct 21, 2008 4:58 am
Location: Melbourne Australia

Wheldall blog what's wrong with What Works C/house

Post by yvonne meyer » Fri Aug 24, 2012 3:21 am

Kevin Wheldall blogs on evidence-based research and the What Works Clearinghouse. Worth reading the entire blog but I have pasted selected quotes below.

What's Wrong with What Works?

http://kevinwheldall.blogspot.com.au/20 ... works.html
Evidence-based practice has become all but a cliché in educational discourse. Perhaps finally tiring of talking about ‘learnings’, ‘privileging’ and verbing any other noun they can get their hands on, educationists have decided to "sing from the same songsheet” of evidence-based practice. That’s got to be a good thing, right? Well, yes, it would be if they were singing the same tune and the same words. Unfortunately, evidence-based practice means different things to different people. This is why I personally prefer the term scientific evidence-based practice. But how are we to know what constitutes (scientific) evidence-based practice?

... My colleagues and I have also been critical of WWC. And not just for being too stringent. Far from being too rigorous, the WWC boffins frequently make, to us, egregious mistakes; mistakes that, far too often for comfort, seem to support a particular approach to teaching and learning...

... Advocates of Direct Instruction (DI) seem to have been particularly ill-served by the methodological ‘rigour’ of WWC, for not only are most more recent studies of the efficacy of DI programs excluded because they do not meet the WWC evidence standards but they also impose a blanket ban on including any study (regardless of technical adequacy) published before 1985; an interesting if somewhat idiosyncratic approach to science. Philip Larkin told us that sex only began in 1963 but who would have thought that there was no educational research worth considering before 1985?...

... If we needed any further proof of the unreliability of WWC reports, we now have their August 2012 report on whether Open Court Reading© improves adolescent literacy (http://tinyurl.com/9nzv5wj). True to form, they discarded 57 out of 58 studies as not meeting evidence standards. On the basis of this one study they concluded that Open Court “was found to have potentially positive effects on comprehension for adolescent readers”. There are at least three problems with this conclusion. First, this is a bold claim based on the results for just one study, the large sample size and their ‘potentially positive’ caveat notwithstanding. Second, the effect size was trivial at 0.16, not even ‘small’, and well below WWC’s own usual threshold of 0.25. Third, and most important of all, this study was not even carried out with adolescents! The study sample comprised “more than 900 first-grade though fifth-grade who attended five schools across the United States”. As Private Eye magazine would have it “shorely shome mishtake” …

... Until we have a large evidence base of methodologically sound randomized control trials on a wide variety of educational programs, methods and procedures, we need a more sophisticated and pragmatic analysis of the evidence we currently have available. It is not a question of accepting any evidence in the absence of good evidence, but rather of assessing the existing research findings and carefully explaining the limitations and caveats.

As I have attempted to show, the spurious rigour of WWC whereby the vast majority of studies on any topic are simply discarded as being too old or too weak methodologically, coupled with their unfortunate habit of making alarming mistakes, makes it hard to trust their judgments. If the suggestions of bias regarding their pedagogical preferences has any substance, we have even more cause for concern. As it stands, What Works simply won’t wash.

volunteer
Posts: 755
Joined: Wed Nov 16, 2011 12:46 pm

Re: Wheldall blog what's wrong with What Works C/house

Post by volunteer » Fri Aug 24, 2012 10:15 am

Thanks Yvonne. I've never found anything useful on it and have always wondered quite how it functioned, so that was very helpful as it made me feel that I was not being unduly over-suspicious about it.

User avatar
maizie
Administrator
Posts: 3121
Joined: Sun Mar 14, 2004 10:38 pm
Location: N.E England

Re: Wheldall blog what's wrong with What Works C/house

Post by maizie » Fri Aug 24, 2012 1:11 pm

Also flagged up on the DDOLL network, along with Wheldall's blog, is this fascinating lecture by Dorothy Bishop:

http://clients2.mediaondemand.net/acamh ... entId=2959#

(It's not immediately obvious that you have to click on the lecture title in order to view it)

Prof. Bishop looks at the problems of using brain scans in research.

User avatar
Susan Godsland
Administrator
Posts: 4973
Joined: Thu Oct 30, 2003 11:10 pm
Location: Exeter UK
Contact:

Re: Wheldall blog what's wrong with What Works C/house

Post by Susan Godsland » Fri Aug 24, 2012 2:06 pm

Prof. Bishop looks at the problems of using brain scans in research.
ahem -4th post down http://www.rrf.org.uk/messageforum/view ... f=1&t=5388 ;-)

User avatar
maizie
Administrator
Posts: 3121
Joined: Sun Mar 14, 2004 10:38 pm
Location: N.E England

Re: Wheldall blog what's wrong with What Works C/house

Post by maizie » Fri Aug 24, 2012 2:53 pm

Oh, Susan. It's the goldfish brain at work...

Anyway, some people might have missed it or forgotten it :oops: (like me...)

User avatar
Susan Godsland
Administrator
Posts: 4973
Joined: Thu Oct 30, 2003 11:10 pm
Location: Exeter UK
Contact:

Re: Wheldall blog what's wrong with What Works C/house

Post by Susan Godsland » Fri Aug 24, 2012 2:57 pm

Goldfish or elephant, if you liked Prof. Bishop's talk you'll like this one too :smile:

Video: Carol Tavris on pseudo-neuroscience

http://www.rrf.org.uk/messageforum/view ... f=1&t=5378

User avatar
palisadesk
Posts: 549
Joined: Sun Mar 20, 2005 2:11 pm

Re: Wheldall blog what's wrong with What Works C/house

Post by palisadesk » Fri Aug 24, 2012 9:45 pm

To get back to what DOES work, enjoy this sample chapter from Explicit Instruction:

http://explicitinstruction.org/download ... hapter.pdf

I've heard the book is excellent, which does not surprise me in the least as Anita Archer is an outstanding presenter, teacher and writer. She is also the author of REWARDS Reading and Phonics for Reading, both very effective programs. I was lucky enough to go to a daylong session on middle-school writing which she instructed. She was amazing.

I didn't get the book when it first came out (too costly) but may try to find it now.

Susan S.

elsiep
Posts: 548
Joined: Tue Nov 02, 2010 10:23 pm

Re: Wheldall blog what's wrong with What Works C/house

Post by elsiep » Sat Aug 25, 2012 10:04 am

There are so many implicit assumptions made in this chapter it's difficult to know where to start. Sorry, but I'm afraid I'm going to have to have a rant.

I can't face going through the sixteen elements of explicit instruction. (Sixteen? The author refers to the capacity of working memory being limited to seven plus or minus two bits of information (although s/he doesn't put it like that) but then expects teachers to bear in mind sixteen elements.) So instead I want to rant about the six principles of effective instruction (p.5).
1. Optimize engaged time/time on task. The more time students are actively participating in
instructional activities, the more they learn.
No, they don't. Learning vs time on task follows an inverted U-shape due to habituation and fatigue. Consolidation of new material takes place (fairly obviously) when the brain isn't focused on absorbing new material - ie during 'downtime'. That's why so many writers, scientists etc spend so much time walking, gardening, painting and so on.
2. Promote high levels of success. The more successful (i.e., correct/accurate) students are when
they engage in an academic task, the more they achieve.
There's some evidence for the benefits of error-free learning in some circumstances but the best predictor of persistence on task is actually variable reinforcement - ie a mix of success and failure. Different people respond differently to different reward schedules.
3. Increase content coverage. The more academic content covered effectively and efficiently, the
greater potential for student learning.
The author hasn't defined 'academic' but otherwise this maxim is self-evident.
4. Have students spend more time in instructional groups. The more time students participate
in teacher-led, skill-level groups versus one-to-one teaching or seatwork activities, the more
instruction they receive, and the more they learn.
Doesn't that depend on a) the child and b) the task? One of the things that bothers me about educational 'methods' is that outcomes are often reported at group level. And there appears to be a growing popularity for meta-analysis. A particular method might be shown to be most effective for most children, and therefore it makes sense to use it in schools; that doesn't mean it is more effective for every child. There are lots of references in this chapter to children with 'learning disabilities' as if they all have a variant of the same condition - not that their learning difficulties might vary widely and require different approaches.
5. Scaffold instruction. Providing support, structure, and guidance during instruction promotes
academic success, and systematic fading of this support encourages students to become more
independent learners.
I 've commented on scaffolding in more detail below.
6. Address different forms of knowledge. The ability to strategically use academic skills and
knowledge often requires students to know different sorts of information at differing levels: the
declarative level (what something is, factual information), the procedural level (how something is
done or performed), and the conditional level (when and where to use the skill).
This is a rather simplistic typology of knowledge and not one based on how the brain works. Unless you derive your understanding of knowledge types from the way the brain processes information you will come unstuck.


Scaffolding. Scaffolding is clearly a key component of explicit instruction but the author doesn't appear to be clear about how or why.

My first introduction to scaffolding came one weekend when my daughter (then 11) had a story to write for homework, but kept saying she couldn't do it and didn't want to talk about why. I was mystified since she spent much of her spare time engaged in creative writing and had written many stories at school. By Sunday afternoon she was distraught and agreed to let me look at what she had to do. I unfolded the paper. The heading said "A plane crash". I immediately saw, in my mind's eye, a tiny silver dot on a vast expanse of sand under a relentless sun. I zoomed in to see three figures huddled in the shade under one crumpled wing of the aircraft. I looked at the first paragraph. It said;

'Here are some words you might find helpful:
mountain blizzard forest avalanche rescue...'

My brain did that thing that brains do when they are expecting coffee and get tea instead; a moment of complete discombobulation as the words provided clashed head-on with my Saharan scenario. I understood why my daughter was so distressed. She also pointed out that she had demonstrated, more times than she could remember, that she could write stories about anything and everything and had always got very good very marks for them. As she put it "They know I can write stories. They know I'll write a good story in the SATs test. Why do I have to keep writing stories? It's a school. I'm supposed to be learning things. Why can't I do something I haven't done hundreds of times before?"

I went to see her teacher. She explained that the useful words were 'scaffolding' and had been shown to be helpful to children who found creative writing challenging. I asked what happened to children who found scaffolding interfered with the creative process. She couldn't tell me.

The author of this chapter explains scaffolding as follows (p.10);
"Scaffolding in an instructional context is analogous to the scaffolding used when
constructing a building. A lot of scaffolding is used as construction begins, but as
the building begins to take shape, the scaffolding is removed in stages until the
building stands on its own. Also, the purpose of scaffolding in both construction
and instruction is the same: to allow individuals to do a task that could not be done
without using it at first. "
This takes today's prize for a hopeless analogy. This person has clearly never built a house. Or even watched one being built. Not carefully, anyway. The first thing a competent housebuilder does is lay foundations. Scaffolding is not required at this point. Indeed it would probably get in the way. Then the lower levels of the house are constructed. Scaffolding is required only when the lower levels reach head height. As the building grows, you need to erect scaffolding so you can reach the upper levels safely. The scaffolding will need to be extended once you start working on the roof. Once the roof is complete, you can start dismantling the scaffolding and once your windows, guttering and external drainage are in place, the scaffolding can be removed completely.

The author of this chapter seems to think that scaffolding is used to support the house whilst it's being constructed. Some houses are indeed erected round a supporting framework, but the framework isn't taken away once the house is complete because if it was, the house would fall down.

I'm aware that the effectiveness of educational scaffolding doesn't depend on a poor analogy, but if the author is ignorant of something as commonplace and important as how people build dwellings, it doesn't inspire confidence (in me at least) that s/he knows what s/he is talking about in other areas.

Why I'm concerned about what I've read about explicit instruction

In order to scientifically test an educational method you do need to test it. It isn't sufficient to cite supporting evidence only; you also need to look at cases where it doesn't work and figure out why. There might be a method you haven't thought of that works even better.

I'm also concerned that there's little evidence of explicit instruction being derived from a good understanding of the biological and cognitive mechanisms of learning - or why they vary across the school population. It look to me like someone has stumbled across a method that's better than any other method available and that's good enough.

I can quite see why explicit instruction could be effective in respect of some knowledge and skills, but to me this sounds like another educational fad, along the lines of Frederick Taylor's scientific management http://en.wikipedia.org/wiki/Scientific_management - and look where that got us.

elsie

User avatar
Debbie Hepplewhite
Administrator
Posts: 3656
Joined: Mon Aug 29, 2005 4:13 pm
Location: Berkshire
Contact:

Re: Wheldall blog what's wrong with What Works C/house

Post by Debbie Hepplewhite » Sat Aug 25, 2012 10:50 am

How ironic - I've just flagged up the sample chapter recommended by Susan S. as very helpful in my work!

I accept, however, that elsep raises issues which challenge various descriptions/findings in the chapter and that her points are not at all invalid.

I have found that one can bring exceptions, challenges, other thoughts to the table, examples which challenge the overarching suggestions - and so on - in almost any scenario.

I suggest that it is possible both to look too deeply into conclusions and guidance and to fail to look deeply enough.

Whatever the subject, there are perspectives which are valid to bring to the table - and examples to show 'otherwise'.

It is fantastic to have on the message forum people who think deeply, have different experiences and who challenge ideas, research - and focus attention on the details - the minutiae - that might be neglected either by design of fault in methodologies, approaches, philosophies - whatever!

It is not sufficient to have 'the research' as one needs to 'research the research' and the body of research.

It is possible to have 'research' where the researchers themselves may not have appreciated their own lack of understanding about all the prevailing conditions - and presume too much about 'developmental stages' versus learners' 'development' when they've had different experiences (be they 'explicitly taught' experiences for example).

There reaches a point, however, when it is surely possible to look at patterns and generalisations based on a broad body of work. I took from the chapter concerned the 'broad' view and related it very much to class teaching.

I am also in the field of supporting practitioners on a large-scale - large-scale in every sense of the word - in terms of cyber-space outreach - and large-scale in the sense of people attending talks and training from many perspectives within the educational domain. What messages can be supportive and effective within a minimal time and with a maximum outreach?

It is possible, and right, for anyone to say at any time, "Ah but... what about these set of circumstances, or this child, or this challenge" and then one is stepping out of the generalisations and giving a view, or describing some similar experience, or offering suggestions... accordingly".

I think the "Ah buts..." are invaluable, valid, and make us all think - also support others to have confidence to bring experiences and exceptions to bear.

I am often an "Ah but...." person myself and that is why I bring guidance and resources to the table which are sometimes a development or a challenge to the accepted practice - but I always make it clear that this is the case. There was the odd conclusion in the example chapter which did not fit into my world-view of education right now - but, in the main, I think it is very helpful and sensible.

elsiep
Posts: 548
Joined: Tue Nov 02, 2010 10:23 pm

Re: Wheldall blog what's wrong with What Works C/house

Post by elsiep » Sat Aug 25, 2012 12:15 pm

Debbie Hepplewhite wrote:How ironic - I've just flagged up the sample chapter recommended by Susan S. as very helpful in my work!
:grin:
I accept, however, that elsep raises issues which challenge various descriptions/findings in the chapter and that her points are not at all invalid.
Thank you, Debbie.
There reaches a point, however, when it is surely possible to look at patterns and generalisations based on a broad body of work. I took from the chapter concerned the 'broad' view and related it very much to class teaching.
I think the theory behind many educational methods is the equivalent of the theory behind medical interventions around 200 years ago.

Take the anecdotal evidence that neat alcohol, applied to the site of a surgical procedure prior to surgery, increased the survival rate of patients. If you'd done a randomised controlled trial in 1812 on whether a good quality brandy, applied to the relevant area of the patient, was effective, you would have found that the brandy group of patients survived better than the non-brandy group (because the alcohol kills microorganisms on the skin, but no one knew that then). But the survival rates wouldn't have been high enough to convince everyone. There were surgeons who thought the brandy was more effective inside the patient than outside. That was because surviving surgical procedures is a multi-factorial problem.

When surgeons started washing their hands (and surgical instruments) between patients, survival rates increased, as they did when anaesthesia became more sophisticated.

Once all these boxes were ticked, post-operative mortality became relatively rare.

The reason it took so long to arrive at a successful approach to major surgery was because not much was known about human physiology or microbial infection. Nowadays, medical interventions are much more likely to start with knowledge about the underlying physiological issues involved, and an intervention worked out from first principles. Interventions need to be tested, but the development of an intervention isn't a hit-and-miss affair like pouring brandy over the patient was.

It looks to me as though explicit instruction is the equivalent of naval surgeons swabbing sailors with brandy prior to amputations, or obstetricians washing their hands between deliveries. Both reduced the mortality rate but no one knew how, or why some patients still died.

Swabbing injured sailors with brandy and getting obstetricians to wash their hands were clearly Good Things, but it didn't mean that either technique was guaranteed to improve the effectiveness of other medical interventions because other factors were involved.

I can't see how education is likely to be effective for all children until and unless it is derived from knowledge about how children learn. We know a great deal about how human beings learn but the people teaching children don't appear to be consulting the people who know how learning works.

User avatar
Debbie Hepplewhite
Administrator
Posts: 3656
Joined: Mon Aug 29, 2005 4:13 pm
Location: Berkshire
Contact:

Re: Wheldall blog what's wrong with What Works C/house

Post by Debbie Hepplewhite » Sat Aug 25, 2012 6:54 pm

I can't see how education is likely to be effective for all children until and unless it is derived from knowledge about how children learn. We know a great deal about how human beings learn but the people teaching children don't appear to be consulting the people who know how learning works.
Who in particular would you recommend? :???:

elsiep
Posts: 548
Joined: Tue Nov 02, 2010 10:23 pm

Re: Wheldall blog what's wrong with What Works C/house

Post by elsiep » Sat Aug 25, 2012 8:42 pm

There are a number of relevant fields of research e.g.;

brain development
cognitive development
sensory processing
conditioning
memory
judgment and decision-making
categorisation
social learning
group interaction etc etc

The point I was making was that people devising educational methods do need to know how human learning works - biologically, psychologically and socially - in the same way as people devising medical interventions need to know how the human body and disease entities work. I'm not seeing this happening in education. Instead fragments of theories are being imported, often misunderstood or taken out of context and often used to justify a pre-existing idea.

elsie

yvonne meyer
Posts: 736
Joined: Tue Oct 21, 2008 4:58 am
Location: Melbourne Australia

Re: Wheldall blog what's wrong with What Works C/house

Post by yvonne meyer » Sun Aug 26, 2012 1:24 am

Why I'm concerned about what I've read about explicit instruction
Elsiep is always concerned about explicit instruction and;

1. looks for flaws in individual research papers which she uses to discount all evidence-based research that supports explicit instruction.

2. is unaware of the multiple, overlapping evidence-based research studies which support explicit instruction.

While decrying the lack of good evidence-based studies and stating that education sorely needs good evidence-based research, Elsiep supports and advocates the child-centred philosophy and (from previous exchanges) the 'child-deficit model' ie, if the child is not learning, there is something wrong with the child, and looks for obscure, unverifiable 'difficulties' with which to label children, none of which have been supported by any evidence-based research studies.

Readers of this forum should be aware that Elsiep comes from a position of support for non-evidence-based child-centred education and take her opinion on evidence that supports explicit instruction with a grain of salt.

JAC
Posts: 517
Joined: Tue Nov 15, 2005 1:51 am

Re: Wheldall blog what's wrong with What Works C/house

Post by JAC » Sun Aug 26, 2012 5:43 am

The point I was making was that people devising educational methods do need to know how human learning works - biologically, psychologically and socially -
..Meanwhile, perhaps a theory of instruction will suffice :smile:
http://psych.athabascau.ca/html/387/Ope ... eory.shtml

elsiep
Posts: 548
Joined: Tue Nov 02, 2010 10:23 pm

Re: Wheldall blog what's wrong with What Works C/house

Post by elsiep » Sun Aug 26, 2012 6:47 am

yvonne meyer wrote:
Why I'm concerned about what I've read about explicit instruction
Elsiep is always concerned about explicit instruction
Elsiep is concerned about all educational 'methods'. That's because different knowledge and skills require different teaching and learning strategies. And different children require different teaching and learning strategies. If you are using a broad, rich curriculum, one 'method' won't work. Also, each child is different and I have yet to see any 'method' that ensures that all children are educated well enough to the point where there are no children 'left behind'. Governments like 'methods' because they hope they will resolve the problem of children being 'left behind' . Teachers are always left to pick up the pieces and teachers and parents get the blame for the 'method' not working. History then repeats itself with a different 'method'. And it all ends in tears. Again.
and;

1. looks for flaws in individual research papers which she uses to discount all evidence-based research that supports explicit instruction.
I do look for flaws in research papers. That is why research papers are published and subjected to peer-review. Flaws are not helpful and most researchers would like to know about them so their research can get better. I'd just like to point out that what I've been finding flaws in in this thread isn't a research paper.
2. is unaware of the multiple, overlapping evidence-based research studies which support explicit instruction.
Yvonne, anybody can find evidence to support their hypothesis; what's more useful (and more scientific) is to test it, ie try to prove it wrong. And to look at cases in which it doesn't work to find out why it doesn't work. That's the essence of the scientific method. I keep reading evidence-based studies which tell me things like the outcome of method A was that test results of 95% of students improved by 10 percentage points whereas with method B the results were no different to controls, therefore there is supporting evidence for method A. Method A is demonstrably better than method B but is 10% on a test actually going to make any difference to all of us in the great scheme of things and what about the 5% of students that don't improve?

That's why I've been wittering on about medical interventions. If you'd done an RCT 150 years ago comparing; pouring brandy on the patient, doctors washing their hands, and using chloroform, against controls, you'd have found that all three interventions raised survival rates. My guess is that doctors washing their hands would have been the front runner by a small margin. So choosing the winning 'method' would have improved patient survival rates. But using all three together would have raised survival rates significantly. And the fact that doctors have maintained a keen interest in why patients didn't survive has meant that our knowledge of antiseptics and anaesthetics has improved considerably. I don't see much research being cited on why 3, 5 or 10% of children don't benefit from brave new methods.
While decrying the lack of good evidence-based studies and stating that education sorely needs good evidence-based research, Elsiep supports and advocates the child-centred philosophy and (from previous exchanges) the 'child-deficit model' ie, if the child is not learning, there is something wrong with the child, and looks for obscure, unverifiable 'difficulties' with which to label children, none of which have been supported by any evidence-based research studies.
I'd opt for a child-centred approach to education over a teacher-centred or curriculum-centred approach because each child is unique, teachers can't know everything and arguments over what should or shouldn't be in the curriculum are interminable. It makes far more sense to equip teachers with a good understanding of how the world works, how all knowledge and skills hang together, and how to help children of different abilities and aptitudes to access the knowledge and skills that optimise those abilities and aptitudes.

As for the 'child-deficit' model - if 27 children in a class of 30 are doing fine on all measures, but 3 are struggling, why would that be? It's unlikely to be poor teaching, or an ineffective educational 'method'. It's far more likely to be that the child has a problem. Are you suggesting that all children's learning difficulties can be alleviated by the right educational 'method'?

I recall a lengthy discussion that you and I had on the TES forum in which I pointed out that the Pediatrics paper http://pediatrics.aappublications.org/c ... 127/3/e818 you kept citing goes into some detail about difficulties that can result from visual processing problems. I even cited page numbers and quoted paragraphs, but you insisted that the possibility that visual impairments could contribute to learning difficulties was 'scotch mist'.
Readers of this forum should be aware that Elsiep comes from a position of support for non-evidence-based child-centred education and take her opinion on evidence that supports explicit instruction with a grain of salt.
Yvonne, I would advocate a child-centred approach to education over a teacher-centred or curriculum-centred one, and I've explained why. That does not mean I would stand back and let children learn everything by exploring for themselves. I have pointed out, repeatedly, that different knowledge and skills and different children need different teaching and learning methods. That's what I mean by child-centred.

As for an evidence-base, it's actually very difficult to design studies that provide evidence, one way or another, about the efficacy of an educational 'approach', as Project Follow Through demonstrates. That, as I've said before, is because an approach is a multi-factorial problem and there are differences of opinion over what constitutes a desirable outcome.

On the subject of Project Follow Through, are there data on long-term outcomes?
Last edited by elsiep on Sun Aug 26, 2012 9:28 am, edited 1 time in total.

Post Reply

Who is online

Users browsing this forum: No registered users and 38 guests