#WorldSTE2013 Conference: Days 3 & 4 (1)

DSC_0218I gave two talks at the WorldSTE2013 Conference.  One discusses some successes and challenges of VVOB‘s SEAL programme.  It relates the programme to Shulman’s Pedagogical Content Knowledge (PCK) and Mishra and Koehler’s extension of Tecnological, Pedagogical and Content Knowledge.  By introducing PCK Shulman aimed at reasserting the importance of content knowledge, as opposed to a sole focus on generic pedagogical skills such as class management that was very much in vogue during the 1980s.  The presentation is embedded below.

The second presentation is based on papers I submitted for MAODE course H810.  It discusses accessibility challenges to (science) education for learners with disabilities in Cambodia. It presents these challenges from an insitutional perspective, applying the Framework of Instutional Change, developed by D.C. North in the 1990s and applied by Ozcan Konur in education.  In particular, it highlights some of slow-changing informal constraints that hamper the effects of changes in formal rules (such as Cambodia’s recent ratification of the UN Convention on the Rights of People with Disabilities) to take much effect in practice.   The framework underlines the importance of aligning formal rules with informal constraints and enforcement characteristics.

On a sidenote, I believe this presentation was about the only one that explicitly discusses inclusive education and how access to science education can be increased for learners with disabilities, despite WHO and the UN estimates that around 15% of learners has some kind of learning disability and that 90% of disabled learners in developing countries do not attend school.


North, D.C. (1994) Institutional Change: A Framework Of Analysis, Economic History, EconWPA, [online] Available from: http://ideas.repec.org/p/wpa/wuwpeh/9412001.html (Accessed 23 December 2012).
Konur, O. (2006) ‘Teaching disabled students in higher education’, Teaching in Higher Education, 11(3), pp. 351–363.
Seale, J. (2006) E-Learning and Disability in Higher Education: Accessibility Research and Practice, Abingdon, Routledge.

#WorldSTE2013 Conference – Day 2: Peer instruction (E. Mazur) and Visible Learning (J.Hattie(

Day 2 of the WorldSTE conference centred on the keynote sessions of two educational ‘rock stars’, Prof. Eric Mazur and Prof. John Hattie.  Both delivered a polished, entertaining presentation, but with little new information for those already familiar with their work.  The conference organizers provided little time for discussion which was, certainly in Hattie’s keynote, a pity.

Mazur’s presentation was a shortened version of the ‘Confessions of a Lecturer’ talk which is available on YouTube in various lengths and colours (recent one).  Concept Tests combined with voting and peer discussion is a powerful way to activate students in lectures.  He referred to Pinker’s ‘curse of knowledge’ as one reason why fellow students are often better than explaining new stuff to each other than lecturers.We have introduced the methodology in Cambodia as well, using voting cards rather electronic clickers.   From my experience, the main challenge for teacher trainers is to get the questions right.  Questions should address a conceptual problem, should preferably relate it to an unfamiliar context and should neither be too easy nor too difficult.

Hattie’s keynote was based on the results of his meta-meta analysis to determine what makes good learning.  It is based on more than 800 meta-analyses into which more than 50 000 individual studies have been integrated.  Starting point are the falsely authoritative claims many teachers and educators make about what works in education, often in conflict with each other.   Extensive reviews of Hattie’s work have been written elsewhere (1, 2).  Here I just write down some personal reflections on his talk:

  1. Hattie likes to unsettle people by listing some of the factors that don’t make a difference, such as teachers’ content knowledge, teacher training, class size, school structures, ability grouping, inquiry-based methods and ICT.  However, I believe that many aspects of teaching quality are interrelated and strengthen or weaken each other.  Content knowledge as such doesn’t make a good teacher, but is a necessary condition for teachers to engage in class discussion or provide meaningful feedback, which are factors that do make a difference in Hattie’s study.  Similarly, class size doesn’t make a difference if  the teacher doesn’t adapt his/her teaching.  However, class size may affect the strategies and possibilities of teachers, as it affects factors such as class management, available space and time. In the same way school structures in itself don’t change teaching quality, but may affect the opportunities for teachers to engage in collaborative lesson preparation, which is strongly endorsed by Hattie.
  2. Similarly, Hattie seemed to admit that many relations are non-linear and that there are threshold effects.  Research on pedagogical content knowledge showed that teachers need to have a good understanding of the concepts they are teaching, but additional specialised subject courses don’t make additional difference.  In Cambodia, limited content knowledge does inhibit teachers to promote deep learning, which also makes a difference in Hattie’s research.

 Overview of effect sizes variables on learning outcomes

Overview of effect sizes variables on learning outcomes

3. This relates to the question how valid results are across countries and cultures.  Hattie’s research is mainly based on research from developed countries and Western cultures, and I wonder how applicable these effect sizes are in other countries and cultures.  The threshold effect size value of 0.4 is based on the typical progression of a student in a developed country.  In a developing country, an effect size of 0.4 may be actually quite high.  Hattie does recognize that the teacher factor is stronger in schools with low-economic status, implying that having a good teacher does matter more for them than for well-off kids.  Banerjee and Duflo have suggested that unlike disappointing results in developed countries ICT may have stronger benefits in developing countries:

“The current view of the use of technology in teaching in the education community is, however, not particularly positive. But this is based mainly on experience from rich countries, where the alternative to being taught by a computer is, to a large extent, being taught by a well-trained and motivated teacher.  This is not always the case in poor countries.  And the evidence from the developing world, though sparse, is quite positive.” (Duflo & Banarjee, Poor Economics,p. 100)

4. Hatie’s research doesn’t take into account factors that lie outside the influence of the school. However, many of the strongest factors in Hattie’s list, such as collaborative lesson preparation and evaluation, class discussions and setting student expectations are well-known for quite some time. Why haven’t they been applied more?  This question has been better addressed by researchers such as North and Konur, who focus on the institutional and organisational analysis of education quality.

5. The concept of effect sizes is statistically shaky.  In a recent paper, Angus Deaton Post writes about effect sizes:

The effect size—the average treatment effect expressed in numbers of standard deviations of the original outcome—though conveniently dimensionless, has little to recommend it. It removes any discipline on what is being compared. Apples and oranges become immediately comparable, as do treatments whose inclusion in a meta-analysis is limited only by the imagination of the analysts in claiming similarity. Beyond that, restrictions on the trial sample will reduce the baseline standard deviation and inflate the effect size. More generally, effect sizes are open to manipulation by exclusion rules. It makes no sense to claim replicability on the basis of effect sizes, let alone to use them to rank projects.

Hattie’s research is wildly ambitious, and therefore a great deal of scrutiny and criticism:

  • sole focus on quantitative research at the expense of qualitative studies (Terhart, 2011, login)
  • statistics underlying effect sizes controversial as well as the premise that effect sizes can be aggregated and compared (blog post on statistics used in Hattie’s research).
  • quality of the studies underlying the meta-analysis varies wildly and shouldn’t simply be aggregated due to publication bias (Higgins and Simpson, 2011, login: an extract

“VL [Visible Learning] seems to suffer from many of the criticisms levelled at meta-analyses and then adds more problems derived from the meta-meta-analysis level. It combines studies across some areas with little apparent conceptual connection; averages results from experimental, nonexperimental, manipulable and non-manipulable studies; effectively ignores subtleties such as implementation cost, additive effects, arbitrary signs and longevity, even when many of the meta-analyses it relies upon carefully highlight these issues. It then combines all the effect sizes by simply adding them together and dividing by the number of studies with no weighting. In this way it develops a simple figure, 0.40, above which, it argues, interventions are ‘worth having’ and below which interventions are not ‘educationally significant’. We argue that the process by which this number has been derived has rendered it effectively meaningless.” (Higgins and Simpson, 2011)

Despite the claim on Hattie’s website, I don’t believe Hattie has finally found the ‘holy grail’ of education research and settled the question of what makes qualitative education. Partly this is due to skepticism whether such a definitive generalized answer across cultures, education levels and economies is possible.  Partly it is due to methodological concerns about the reliability of aggregating aggregations of effect sizes and the validity of excluding qualitative research and all factors that lie outside the influence of the school.

Finally, the Hattie keynote made me nostalgic about the H809 course in MAODE during which papers would be turned inside out until you would be convinced that each constituted the worst kind of educational research ever conducted.  Hattie’s research would fit excellently in such a context.

#WorldSTE2013 Conference – Day 1

icase2013I’m participating (and blogging) this week from the World Conference in Science and Technology Education (Worldste2013) in Sarawak, Borneo (Malaysia). Day 1 got off to quite a chaotic start with long queues at registration desks, messing up the session schedule resulting in some people presenting in front of empty chairs.  Gradually, the dust settled a bit and I hope everything goes a bit more smoothly tomorrow.

The dominating topic during the first day was the alleged low interest of pupils and students in many countries in science and, as a result, insufficient numbers pursuing higher education in science.  Recipes to deal with this focused on:

  • Increase the authentic character of science education, focusing on topics students find interesting.  Examples include climate science, renewable energy, future health, colonisation of Mars and sustainable housing.
  • The role of technology in improving the quality of science education, mentioning recent developments such as MOOCs (wrongfully crediting Stanford-startup Udacity with its origin), the flipped classroom and the SAMR model to describe the potential use of ICT from merely substituting to transforming classroom practice.
  • Underlining the importance of science education in addressing global challenges such as climate change, biodiversity and population growth. Dr. Hubert Gijzen,  UNESCO chief from the Regional Science Bureau for Asia and Pacific, called for a stronger focus on sustainability and peace and security, next to combatting poverty.

I wonder whether increasing relevance and authenticity is the (only) recipe to convince more students to study science.  Plenty of students study accounting or marketing without any engaging projects.  Second, these calls for authentic science are hardly new.  However, research shows that initiative after initiative don’t yet seem to be having much impact.  The HBO series ‘Breaking Bad’ seems to have a much stronger effect on interest in chemistry studies in Flanders than authentic science projects .  Also, calls for more science graduates seem not backed up by unemployment figures (article in New Scientist).  Rather, it can be argued that there are not enough graduates in specific fields of STEM (e.g. biotech, teaching) and that the quality of science education can be improved in order to achieve higher levels of numeracy and scientific literacy.

Dr. Young Choi provided an overview of the UNESCO-supported Asia Green School project, an extracurricular programme to help primary and secondary school teachers with translating environmental sustainability into their schools.  I found the most interesting aspect in this the involvement of the local and central government and private actors and the setting up of green schools communities of practice.

The STELR project was presented by Alan Finkel and Peter Pentland as an attempt to help teachers engaging in more authentic projects with their studentsThe focus was on alignment with the curriculum and promoting inquiry-based approaches.  An interesting element is the development of career profiles to show students what careers are possible with a STEM education and what kind of skills and knowledge are required for them.  Unfortunately, resources are not openly accessible and therefore hardly useful outside the mostly Australian schools that participate in the programme.

Outside the keynotes an interesting presentation came from Dr. Gregory Smith (Charles Darwin Un., Australia) who presented a study that examined how primary students’ perceptions of science classroom practice evolved from 2001 to 2011.  Despite a variety of research, recommendations and programmes the study found little change in students engaging in the active exploration of phenomena, ideas and relevant science questions or the use of open investigations. However, the nature of the classroom was found to be trending towards being more collaborative and there was a strong increase in the use of ICT and internet.  It’s a study based on a questionnaire and simple non-parametric chi-square analyses, which could serve as an example to investigate student teachers’ perceptions of lesson practice in Cambodia.