Day 2 of the WorldSTE conference centred on the keynote sessions of two educational ‘rock stars’, Prof. Eric Mazur and Prof. John Hattie. Both delivered a polished, entertaining presentation, but with little new information for those already familiar with their work. The conference organizers provided little time for discussion which was, certainly in Hattie’s keynote, a pity.
Mazur’s presentation was a shortened version of the ‘Confessions of a Lecturer’ talk which is available on YouTube in various lengths and colours (recent one). Concept Tests combined with voting and peer discussion is a powerful way to activate students in lectures. He referred to Pinker’s ‘curse of knowledge’ as one reason why fellow students are often better than explaining new stuff to each other than lecturers.We have introduced the methodology in Cambodia as well, using voting cards rather electronic clickers. From my experience, the main challenge for teacher trainers is to get the questions right. Questions should address a conceptual problem, should preferably relate it to an unfamiliar context and should neither be too easy nor too difficult.
Hattie’s keynote was based on the results of his meta-meta analysis to determine what makes good learning. It is based on more than 800 meta-analyses into which more than 50 000 individual studies have been integrated. Starting point are the falsely authoritative claims many teachers and educators make about what works in education, often in conflict with each other. Extensive reviews of Hattie’s work have been written elsewhere (1, 2). Here I just write down some personal reflections on his talk:
- Hattie likes to unsettle people by listing some of the factors that don’t make a difference, such as teachers’ content knowledge, teacher training, class size, school structures, ability grouping, inquiry-based methods and ICT. However, I believe that many aspects of teaching quality are interrelated and strengthen or weaken each other. Content knowledge as such doesn’t make a good teacher, but is a necessary condition for teachers to engage in class discussion or provide meaningful feedback, which are factors that do make a difference in Hattie’s study. Similarly, class size doesn’t make a difference if the teacher doesn’t adapt his/her teaching. However, class size may affect the strategies and possibilities of teachers, as it affects factors such as class management, available space and time. In the same way school structures in itself don’t change teaching quality, but may affect the opportunities for teachers to engage in collaborative lesson preparation, which is strongly endorsed by Hattie.
- Similarly, Hattie seemed to admit that many relations are non-linear and that there are threshold effects. Research on pedagogical content knowledge showed that teachers need to have a good understanding of the concepts they are teaching, but additional specialised subject courses don’t make additional difference. In Cambodia, limited content knowledge does inhibit teachers to promote deep learning, which also makes a difference in Hattie’s research.
Overview of effect sizes variables on learning outcomes
3. This relates to the question how valid results are across countries and cultures. Hattie’s research is mainly based on research from developed countries and Western cultures, and I wonder how applicable these effect sizes are in other countries and cultures. The threshold effect size value of 0.4 is based on the typical progression of a student in a developed country. In a developing country, an effect size of 0.4 may be actually quite high. Hattie does recognize that the teacher factor is stronger in schools with low-economic status, implying that having a good teacher does matter more for them than for well-off kids. Banerjee and Duflo have suggested that unlike disappointing results in developed countries ICT may have stronger benefits in developing countries:
“The current view of the use of technology in teaching in the education community is, however, not particularly positive. But this is based mainly on experience from rich countries, where the alternative to being taught by a computer is, to a large extent, being taught by a well-trained and motivated teacher. This is not always the case in poor countries. And the evidence from the developing world, though sparse, is quite positive.” (Duflo & Banarjee, Poor Economics,p. 100)
4. Hatie’s research doesn’t take into account factors that lie outside the influence of the school. However, many of the strongest factors in Hattie’s list, such as collaborative lesson preparation and evaluation, class discussions and setting student expectations are well-known for quite some time. Why haven’t they been applied more? This question has been better addressed by researchers such as North and Konur, who focus on the institutional and organisational analysis of education quality.
5. The concept of effect sizes is statistically shaky. In a recent paper, Angus Deaton Post writes about effect sizes:
The effect size—the average treatment effect expressed in numbers of standard deviations of the original outcome—though conveniently dimensionless, has little to recommend it. It removes any discipline on what is being compared. Apples and oranges become immediately comparable, as do treatments whose inclusion in a meta-analysis is limited only by the imagination of the analysts in claiming similarity. Beyond that, restrictions on the trial sample will reduce the baseline standard deviation and inflate the effect size. More generally, effect sizes are open to manipulation by exclusion rules. It makes no sense to claim replicability on the basis of effect sizes, let alone to use them to rank projects.
Hattie’s research is wildly ambitious, and therefore a great deal of scrutiny and criticism:
- sole focus on quantitative research at the expense of qualitative studies (Terhart, 2011, login)
- statistics underlying effect sizes controversial as well as the premise that effect sizes can be aggregated and compared (blog post on statistics used in Hattie’s research).
- quality of the studies underlying the meta-analysis varies wildly and shouldn’t simply be aggregated due to publication bias (Higgins and Simpson, 2011, login: an extract
“VL [Visible Learning] seems to suffer from many of the criticisms levelled at meta-analyses and then adds more problems derived from the meta-meta-analysis level. It combines studies across some areas with little apparent conceptual connection; averages results from experimental, nonexperimental, manipulable and non-manipulable studies; effectively ignores subtleties such as implementation cost, additive effects, arbitrary signs and longevity, even when many of the meta-analyses it relies upon carefully highlight these issues. It then combines all the effect sizes by simply adding them together and dividing by the number of studies with no weighting. In this way it develops a simple figure, 0.40, above which, it argues, interventions are ‘worth having’ and below which interventions are not ‘educationally significant’. We argue that the process by which this number has been derived has rendered it effectively meaningless.” (Higgins and Simpson, 2011)
Despite the claim on Hattie’s website, I don’t believe Hattie has finally found the ‘holy grail’ of education research and settled the question of what makes qualitative education. Partly this is due to skepticism whether such a definitive generalized answer across cultures, education levels and economies is possible. Partly it is due to methodological concerns about the reliability of aggregating aggregations of effect sizes and the validity of excluding qualitative research and all factors that lie outside the influence of the school.
Finally, the Hattie keynote made me nostalgic about the H809 course in MAODE during which papers would be turned inside out until you would be convinced that each constituted the worst kind of educational research ever conducted. Hattie’s research would fit excellently in such a context.