#WorldSTE2013 Conference: Days 3 & 4 (1)

DSC_0218I gave two talks at the WorldSTE2013 Conference.  One discusses some successes and challenges of VVOB‘s SEAL programme.  It relates the programme to Shulman’s Pedagogical Content Knowledge (PCK) and Mishra and Koehler’s extension of Tecnological, Pedagogical and Content Knowledge.  By introducing PCK Shulman aimed at reasserting the importance of content knowledge, as opposed to a sole focus on generic pedagogical skills such as class management that was very much in vogue during the 1980s.  The presentation is embedded below.

The second presentation is based on papers I submitted for MAODE course H810.  It discusses accessibility challenges to (science) education for learners with disabilities in Cambodia. It presents these challenges from an insitutional perspective, applying the Framework of Instutional Change, developed by D.C. North in the 1990s and applied by Ozcan Konur in education.  In particular, it highlights some of slow-changing informal constraints that hamper the effects of changes in formal rules (such as Cambodia’s recent ratification of the UN Convention on the Rights of People with Disabilities) to take much effect in practice.   The framework underlines the importance of aligning formal rules with informal constraints and enforcement characteristics.

On a sidenote, I believe this presentation was about the only one that explicitly discusses inclusive education and how access to science education can be increased for learners with disabilities, despite WHO and the UN estimates that around 15% of learners has some kind of learning disability and that 90% of disabled learners in developing countries do not attend school.

References

North, D.C. (1994) Institutional Change: A Framework Of Analysis, Economic History, EconWPA, [online] Available from: http://ideas.repec.org/p/wpa/wuwpeh/9412001.html (Accessed 23 December 2012).
Konur, O. (2006) ‘Teaching disabled students in higher education’, Teaching in Higher Education, 11(3), pp. 351–363.
Seale, J. (2006) E-Learning and Disability in Higher Education: Accessibility Research and Practice, Abingdon, Routledge.

#WorldSTE2013 Conference – Day 2: Peer instruction (E. Mazur) and Visible Learning (J.Hattie(

Day 2 of the WorldSTE conference centred on the keynote sessions of two educational ‘rock stars’, Prof. Eric Mazur and Prof. John Hattie.  Both delivered a polished, entertaining presentation, but with little new information for those already familiar with their work.  The conference organizers provided little time for discussion which was, certainly in Hattie’s keynote, a pity.

Mazur’s presentation was a shortened version of the ‘Confessions of a Lecturer’ talk which is available on YouTube in various lengths and colours (recent one).  Concept Tests combined with voting and peer discussion is a powerful way to activate students in lectures.  He referred to Pinker’s ‘curse of knowledge’ as one reason why fellow students are often better than explaining new stuff to each other than lecturers.We have introduced the methodology in Cambodia as well, using voting cards rather electronic clickers.   From my experience, the main challenge for teacher trainers is to get the questions right.  Questions should address a conceptual problem, should preferably relate it to an unfamiliar context and should neither be too easy nor too difficult.

Hattie’s keynote was based on the results of his meta-meta analysis to determine what makes good learning.  It is based on more than 800 meta-analyses into which more than 50 000 individual studies have been integrated.  Starting point are the falsely authoritative claims many teachers and educators make about what works in education, often in conflict with each other.   Extensive reviews of Hattie’s work have been written elsewhere (1, 2).  Here I just write down some personal reflections on his talk:

  1. Hattie likes to unsettle people by listing some of the factors that don’t make a difference, such as teachers’ content knowledge, teacher training, class size, school structures, ability grouping, inquiry-based methods and ICT.  However, I believe that many aspects of teaching quality are interrelated and strengthen or weaken each other.  Content knowledge as such doesn’t make a good teacher, but is a necessary condition for teachers to engage in class discussion or provide meaningful feedback, which are factors that do make a difference in Hattie’s study.  Similarly, class size doesn’t make a difference if  the teacher doesn’t adapt his/her teaching.  However, class size may affect the strategies and possibilities of teachers, as it affects factors such as class management, available space and time. In the same way school structures in itself don’t change teaching quality, but may affect the opportunities for teachers to engage in collaborative lesson preparation, which is strongly endorsed by Hattie.
  2. Similarly, Hattie seemed to admit that many relations are non-linear and that there are threshold effects.  Research on pedagogical content knowledge showed that teachers need to have a good understanding of the concepts they are teaching, but additional specialised subject courses don’t make additional difference.  In Cambodia, limited content knowledge does inhibit teachers to promote deep learning, which also makes a difference in Hattie’s research.

 Overview of effect sizes variables on learning outcomes

Overview of effect sizes variables on learning outcomes

3. This relates to the question how valid results are across countries and cultures.  Hattie’s research is mainly based on research from developed countries and Western cultures, and I wonder how applicable these effect sizes are in other countries and cultures.  The threshold effect size value of 0.4 is based on the typical progression of a student in a developed country.  In a developing country, an effect size of 0.4 may be actually quite high.  Hattie does recognize that the teacher factor is stronger in schools with low-economic status, implying that having a good teacher does matter more for them than for well-off kids.  Banerjee and Duflo have suggested that unlike disappointing results in developed countries ICT may have stronger benefits in developing countries:

“The current view of the use of technology in teaching in the education community is, however, not particularly positive. But this is based mainly on experience from rich countries, where the alternative to being taught by a computer is, to a large extent, being taught by a well-trained and motivated teacher.  This is not always the case in poor countries.  And the evidence from the developing world, though sparse, is quite positive.” (Duflo & Banarjee, Poor Economics,p. 100)

4. Hatie’s research doesn’t take into account factors that lie outside the influence of the school. However, many of the strongest factors in Hattie’s list, such as collaborative lesson preparation and evaluation, class discussions and setting student expectations are well-known for quite some time. Why haven’t they been applied more?  This question has been better addressed by researchers such as North and Konur, who focus on the institutional and organisational analysis of education quality.

5. The concept of effect sizes is statistically shaky.  In a recent paper, Angus Deaton Post writes about effect sizes:

The effect size—the average treatment effect expressed in numbers of standard deviations of the original outcome—though conveniently dimensionless, has little to recommend it. It removes any discipline on what is being compared. Apples and oranges become immediately comparable, as do treatments whose inclusion in a meta-analysis is limited only by the imagination of the analysts in claiming similarity. Beyond that, restrictions on the trial sample will reduce the baseline standard deviation and inflate the effect size. More generally, effect sizes are open to manipulation by exclusion rules. It makes no sense to claim replicability on the basis of effect sizes, let alone to use them to rank projects.

Hattie’s research is wildly ambitious, and therefore a great deal of scrutiny and criticism:

  • sole focus on quantitative research at the expense of qualitative studies (Terhart, 2011, login)
  • statistics underlying effect sizes controversial as well as the premise that effect sizes can be aggregated and compared (blog post on statistics used in Hattie’s research).
  • quality of the studies underlying the meta-analysis varies wildly and shouldn’t simply be aggregated due to publication bias (Higgins and Simpson, 2011, login: an extract

“VL [Visible Learning] seems to suffer from many of the criticisms levelled at meta-analyses and then adds more problems derived from the meta-meta-analysis level. It combines studies across some areas with little apparent conceptual connection; averages results from experimental, nonexperimental, manipulable and non-manipulable studies; effectively ignores subtleties such as implementation cost, additive effects, arbitrary signs and longevity, even when many of the meta-analyses it relies upon carefully highlight these issues. It then combines all the effect sizes by simply adding them together and dividing by the number of studies with no weighting. In this way it develops a simple figure, 0.40, above which, it argues, interventions are ‘worth having’ and below which interventions are not ‘educationally significant’. We argue that the process by which this number has been derived has rendered it effectively meaningless.” (Higgins and Simpson, 2011)

Despite the claim on Hattie’s website, I don’t believe Hattie has finally found the ‘holy grail’ of education research and settled the question of what makes qualitative education. Partly this is due to skepticism whether such a definitive generalized answer across cultures, education levels and economies is possible.  Partly it is due to methodological concerns about the reliability of aggregating aggregations of effect sizes and the validity of excluding qualitative research and all factors that lie outside the influence of the school.

Finally, the Hattie keynote made me nostalgic about the H809 course in MAODE during which papers would be turned inside out until you would be convinced that each constituted the worst kind of educational research ever conducted.  Hattie’s research would fit excellently in such a context.

#WorldSTE2013 Conference – Day 1

icase2013I’m participating (and blogging) this week from the World Conference in Science and Technology Education (Worldste2013) in Sarawak, Borneo (Malaysia). Day 1 got off to quite a chaotic start with long queues at registration desks, messing up the session schedule resulting in some people presenting in front of empty chairs.  Gradually, the dust settled a bit and I hope everything goes a bit more smoothly tomorrow.

The dominating topic during the first day was the alleged low interest of pupils and students in many countries in science and, as a result, insufficient numbers pursuing higher education in science.  Recipes to deal with this focused on:

  • Increase the authentic character of science education, focusing on topics students find interesting.  Examples include climate science, renewable energy, future health, colonisation of Mars and sustainable housing.
  • The role of technology in improving the quality of science education, mentioning recent developments such as MOOCs (wrongfully crediting Stanford-startup Udacity with its origin), the flipped classroom and the SAMR model to describe the potential use of ICT from merely substituting to transforming classroom practice.
  • Underlining the importance of science education in addressing global challenges such as climate change, biodiversity and population growth. Dr. Hubert Gijzen,  UNESCO chief from the Regional Science Bureau for Asia and Pacific, called for a stronger focus on sustainability and peace and security, next to combatting poverty.

I wonder whether increasing relevance and authenticity is the (only) recipe to convince more students to study science.  Plenty of students study accounting or marketing without any engaging projects.  Second, these calls for authentic science are hardly new.  However, research shows that initiative after initiative don’t yet seem to be having much impact.  The HBO series ‘Breaking Bad’ seems to have a much stronger effect on interest in chemistry studies in Flanders than authentic science projects .  Also, calls for more science graduates seem not backed up by unemployment figures (article in New Scientist).  Rather, it can be argued that there are not enough graduates in specific fields of STEM (e.g. biotech, teaching) and that the quality of science education can be improved in order to achieve higher levels of numeracy and scientific literacy.

Dr. Young Choi provided an overview of the UNESCO-supported Asia Green School project, an extracurricular programme to help primary and secondary school teachers with translating environmental sustainability into their schools.  I found the most interesting aspect in this the involvement of the local and central government and private actors and the setting up of green schools communities of practice.

The STELR project was presented by Alan Finkel and Peter Pentland as an attempt to help teachers engaging in more authentic projects with their studentsThe focus was on alignment with the curriculum and promoting inquiry-based approaches.  An interesting element is the development of career profiles to show students what careers are possible with a STEM education and what kind of skills and knowledge are required for them.  Unfortunately, resources are not openly accessible and therefore hardly useful outside the mostly Australian schools that participate in the programme.

Outside the keynotes an interesting presentation came from Dr. Gregory Smith (Charles Darwin Un., Australia) who presented a study that examined how primary students’ perceptions of science classroom practice evolved from 2001 to 2011.  Despite a variety of research, recommendations and programmes the study found little change in students engaging in the active exploration of phenomena, ideas and relevant science questions or the use of open investigations. However, the nature of the classroom was found to be trending towards being more collaborative and there was a strong increase in the use of ICT and internet.  It’s a study based on a questionnaire and simple non-parametric chi-square analyses, which could serve as an example to investigate student teachers’ perceptions of lesson practice in Cambodia.

 

Too Hard To Measure: On the Value of Experiments and the Difficulty to Measure Lesson Quality

Interesting article in The Guardian (from some time ago, I’m a slow reader) about the overblown importance attributed to doing experiments during science lessons.

The article reminds me of my experience in Cambodia, where experiments are also frequently espoused as proof of a student-centred lesson.  In reality experiments in Cambodian classrooms are often a very teacher-centred activity:

  • the teacher demonstrates and students (at best) trying to observe what happens.
  • students do the experiment in large groups, by adhering to a strict series of steps outlined in a worksheet.
  • students work in large groups, in which usually only one or two students do the work, The others are merely bystanders.
  • the procedure, observations and interpretation of the experiment are laid down in detail beforehand.

The article touches upon two interesting elements.  First, there is the questionable educational value of many experiments in science classes.  secondly, there is the challenge to measure lesson quality beyond ‘ticking off’ the occurrence of activities such as experiments.

The article refers to ‘The Fallacy of Induction‘ from Rosalind Driver.  Her book ‘Making Sense of Secondary Science’ is an excellent book on misconceptions in science education and has been an important inspiration for me.  

“Driver doesn’t dismiss practical work in science, but argues that ‘Many pupils do not know the purpose of practical activity, thinking that they ‘do experiments’ in school to see if something works, rather than to reflect on how a theory can explain observations.” (Driver et al, 1993, p.7).

She raises two main arguments.  First, practical activities are often presented to students as a simulation of ‘how science really works’, collecting data, making observations, drawing inferences and arriving at a conclusion which is the accepted explanation.  It’s simplistic, and pupils happily play along, following the ‘recipe’ in the ‘cookbook’, checking whether they have ‘the right answer’.  In 

reality, science rarely works this way:

“For a long time philosophers of science and scientists themselves have recognised the limitations of the inductive position and have acknowledged the important role that imagination plays in the construction of scientific theories.” (Driver, 1994, p.43)

The second argument is that pupils don’t arrive in class with a blank slate, but with a whole range of self-constructed interpretations or ‘theories’ on how natural phenomena work. These ‘preconceptions’ require more than an experiment to change, as children tend to fit observations within their own ‘theoretical framework’.

Observations are not longer seen as objective but influenced by the theoretical perspective of the observer. ‘As Popper said, ‘we are prisoners caught in the framework of our theories.’ This too has implications for school science, for children, too, can be imprisoned in this way by their preconceptions, observing the world throught their own particular ‘conceptual spectacles.’ (Driver, 1994, p.44)

“Misconceptions can be changed if they are made explicit, discussed and challenged with contradicting evidence.  After this ‘unlearning’ phase, children may adopt a different framework.  Driver concludes: ‘Experience by itself is not enough. It is the sense that students make of it that matters” (Driver et al, 1993, p.7).  

Discussion activities, in which pupils have the opportunity to make their reasoning explicit and to engage with and try out alternative viewpoints, including the ‘scientific one’, need to be central (cognitive conflict). Practical activities can be complementary to these discussions, instead of the other way around, when discussion and conclusion are quickly reeled off at the end of the practicum.

 

Measuring lesson quality

However, the love for experiments while neglecting the question whether and what students are actually learning also touches upon the difficulty to measure adequately lesson quality.  Limited time and resources result in a focus on outward and visible signs. However, these:

  • deny the complexity of teaching and learning;
  • deny the individuality of students’ learning and understanding;
  • steers teachers and programme staff towards focusing on these outward signs, as they know they will be evaluated on these criteria. 

Collecting valid and reliable data on lesson quality is hard.  Self-assessment instruments are notoriously prone to confirmation bias. Lesson observations don’t give a reliable everyday picture of lesson practice.  They suffer from the fact that teachers pull out special lessons when visitors appear for announced (or unannounced) visits.   Conversely, as Cuban describes beautifully, other teachers tremble and panic when an evaluator walks into their classroom and the lesson becomes a shambles.

Evidence-based evaluation is often touted as the way forward for development projects.  Randomized trials in health have been useful to collect a body of knowledge on what works and what not. In a randomized trial a group of students where teachers received pedagogical training is compared with a group of students where teachers didn’t receive training.  Comparisons can be made with test scores, student satisfaction or drop-outs.


However, test scores are unsuitable as exams are notoriously prone to cheating and questions focus on recollecting factual knowledge, the opposite of what we want to achieve.  A self-designed test could be a solution, but there’s the risk that programme activities will focus more on the test than on improving teaching skills.  Student satisfaction scores are prone to the aforementioned confirmation bias.  Drop-outs are hard to use as they are influenced by many interrelated factors such as geography, economic growth and government policy.


Ownership by the direct target group on the evaluation is part of the solution in my opinion, as well as using a variety of data sources.  In future blog posts I plan to write more on how we try to measure lesson quality.


————————

For more detail see this available study from Prof. James Dillon (pdf) on the value of practical work in science education.
Dri­ver, R. (1994) ‘The fal­lacy of induc­tion in sci­ence teach­ing’, in Teach­ing Sci­ence, ed. Levin­son, R., Lon­don, Rout­ledge, pp.41–48.

Driver, R., Squires, A., Rushworth, P. and Wood-Robinson, V. (1993) Making Sense of Secondary Science, Routledge.

Changing Physics Education in Cambodia: Beyond the Workshop

Last week saw the organisation of a workshop on physics education for teacher trainers in Cambodia at the regional teacher training centre in Kandal province.  All Cambodian physics teacher trainers were present, what makes around 20 people.  The workshop lasted 5 days.   Each day we discussed a different part from the curriculum.  There were days we focused on sound, mechanics, pressure, optics and electricity and magnetism.  The last day participants collaboratively made a lesson plan using materials they’d learned.   There was a strong emphasis on low-cost experiments, but also attention for simulations and animations and student-centred approaches.  

The concept underlying the workshop – and actually the whole programme – is the TPACK concept (Mishra and Koehler,2006; Koehler and Mishra, 2007; Abbitt, 2011), an extension of Shulman’s idea of pedagogical content knowledge.  This is knowledge of pedagogy that is applicable to the teaching of specific content.  TPACK extends this idea with technologies.  The core idea of TPACK is that the use of technologies in education – and in Cambodia analogous technologies such as experiments, posters or cards play a much larger role than digital technologies – should be considered in relation to content and pedagogy.  Just using an experiment or an animation just for the sake of it, without thinking about how it will make your lesson better is not useful. This may seem obvious but many interventions seem to do just this, introducing certain technologies (blogging, wikis…) or pedagogies (concept mapping, learner-centred methodologies…) without detailed consideration of the curriculum content teachers actually have to cover.

 

The workshop is the result of three years of preparatory work with a wonderful team of teachers and teacher trainers from the college in Kandal.  Since 2008 we’ve worked with them to select materials and activities for those curriculum topics they found most challenging, try them out in their lessons, develop accessible manuals and short experiment videos (See for example this experiment video on toilet rolls and pressure) and learn to facilitate the activities themselves. 

Manuals have been officially approved by the Cambodian Ministry of Education, an important milestone in Cambodia, as it means that they can be distributed and endorsed nation-wide.  Although we do hope that these manuals by themselves are inviting, an official stamp of approval is likely to act as an extra stimulation.  It’s great to see teacher trainers themselves facilitate the workshop without much involvement of us.  Above all, they enjoy it as well to explain all these experiments and activities to their colleagues as well.

The downside of involving all stakeholders is a very long development cycle.  Getting from a first selection of content until the final, approved product has taken us several years.   Having a first edition published sooner would have enabled us to envisage a second edition within the programme lifetime.  

However, our objective is not to organize great workshops, but to improve science teaching.   Whether our workshops will have a strong effect on the ground remains to be seen.  There are quite a few hurdles between a good workshop and improved learning by grade 7-9 pupils.  Teacher trainers may feel insufficiently comfortable with the materials to use them, support from college management may lack, an overloaded curriculum and recalling-based assessment may favour rote learning.  Student teachers may misunderstand techniques, fail to see any benefits or be discouraged by their school environment.

Targeting teacher trainers has been a deliberate decision.  As they teach future teachers the potential impact is very high.  However, the adopted cascading strategy bears the risk of a watering down the content.    Measuring impact is notoriously difficult, perhaps even more so in Asia, where stated preference methods are prone to response and cultural bias.

Despite continuous M&E efforts we don’t have a clear insight yet into the impact of our activities at teacher training level on the pupils.  The main reasons are the fact that measuring impact is time intensive, that an observable impact may take time to manifest and that a clear impact of the programme within the messy complexity of teaching and learning in a crowded donor landscape is hard to distinguish.
References
Abbitt, J.T. (2011) ‘Measuring Technological Pedagogical Content Knowledge in Preservice Teacher Education: A Review of Current Methods and Instruments’, Journal of Research on Technology in Education, 43(4).
Koehler, Matthew J and Mishra, Punya (2005) ‘Teachers learning technology by design’, Journal of Computing in Teacher Education, 21(3), pp. 94–102.
Mishra, Punya and Koehler, Matthew J. (2006) ‘Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge’, Teachers College Record, 108(6), pp. 1017–1054.

Thinking positively

In Cambodia VVOB focuses on student-centered approaches in science education in teacher training institutes.  Teacher trainers are guided towards adopting approaches to teaching that involve students, stimulate them to think and engage instead of passively noting down and regurgitating what the teacher declaims.

Some student-centred approaches focus on stimulating students’ writing skills.  Writing and the skills that accompany it (collecting thoughts, filtering, structuring, creativity, conciseness…) are arguably important 21st century skills.  To extend writing in science lessons beyond copying teachers’ notes we introduced techniques such as 3-2-1 sheets, 2-minute papers and creative writing (link to my presentation on these).  These techniques aim at stimulating students to write about what they have learned in their own words.

There may be unintended benefits as well

Interestingly, New Scientist refers to a study from David Creswell from Carnegie Mellon University that revealed that letting students write about what they have learned, stimulating qualities such as creativity and independence, gives them ‘self-affirmation’ that enables them to perform better.  

‘Compared with a control group, students who ‘self-affirmed’ in this way had lower levels of adrenaline and other fight-or-flight hormones in their urine on exam day.’  (Health Psychology, vol 28, p 554, quoted in New Scientist, 27 August 2011)

Apart from letting students re-interpret what they have learned, these writing exercises may also – if well designed – contribute to improving students’ sense of self-worth.