The Edutech Debate and the World Bank Blog from Michael Trucano regularly provide excellent background reading on the effectiveness on ICT in education (‘computers in the classroom’). Discussion surged again with the publication of an Inter-American Development Bank (IDB) evaluation of the One Laptop Per Child (OLPC) project in Peru, which sparked a lengthy but worthwhile discussion on the EduTech website.
The IDB study on the OLPC programme in Peru did not find any tangible effect of the programme on students’ test results (national assessments in Math and language test results) 15 months after implementation (Link to Economist article on the study).
The IDB applied a randomized control experiment by sampling five students per grade per school out of 320 schools (2/3 of which were in a treatment receiving the intervention) at intervals of 3 and 15 months. The recently published assessment brief covered the 15 month data.
Some people were quick to point to the impact of the programme beyond the immediate effect on test results. Nicholas Negroponte, founder and chairman of the OLPC non-profit stressed that the purpose of OLPC was not to improve classroom learning only, but learning in the child’s whole life. The computers arguably improved reading comprehension, parent involvement, critical and creative thinking, initiative and discovery. Also the OLPC Peru team defends the project and points out that attitudes and expectations of students, parents and teachers have changed (within 15 months of implementation of the project). According to project leader Oscar Bacerro (reply to Economist and article in EduTech Debate) the Peru programme the educational system in such poor shape that improving the quality of its teachers would require 10-15 years. He focuses on the lack of a fertile environment in the Peruvian educational system to introduce the programme.
“It was clear to us the main challenge for our project would not be “teacher training” on how to use computers in the classroom because most of our teachers needed exceedingly much more than ICT literacy courses.”
He argues that many teachers lack basic numeracy and literacy skills. Notwithstanding the fact that programme design included 40 hours of training for teachers, this was clearly insufficient for the majority of teachers. The IDB analysis showed that almost half the students were prohibited from taking the XOs home and that half of all teachers didn’t use the devices in the classroom. Changes in cognitive abilities take many years to materialize. Is this a case of too much emphasis on the infrastructure and not enough attention on technical and pedagogical support?
The OLPC project was not implemented in isolation. There were several interventions aimed at improving the pedagogical and institutional framework of the schools. A large in-service training programme was included in the programme design. The OLPC part, of course, gets all the media attention. I’m not sure what percentage of the budget went to infrastructure and what part to capacity development of teachers and educational stakeholders. In the VVOB programme in Cambodia the share of infrastructure is less than 20% which might even be too high.
Stanford professor Larry Cuban has been a famous sceptic of technology in the classroom since many years. Cuban’s core point is that school improvement is hard and at its core is not about technology. In fact, it’s the organization’s skill at defining a shared vision, communicating, collaborating, evaluating, changing, etc. that is the driver of effective outcomes.
I do not doubt the valuable role ICT can play in education. I fully believe that technology can improve access to learning and its quality. However, I also believe that ICT in education often is about the least cost efficient way to improve education – with some notable exceptions. Toyama refers to the difference between the purchase cost of a laptop and the total operation cost (TOC), which includes maintenance, electricity, software and connectivity. The TOC is usually 5 – 10 times higher than the purchase cost, meaning that a 300 USD computer represents an investment of approx. 300 USD per year (assuming a generous 5 year computer lifetime). This is quite a lot, taking into account that the Peruvian government spends on average 686 USD per child in primary education and 782 USD in secondary education – in Cambodia it is 54 USD/ child. The question is whether this amount could not be better spent on improvements in teacher education, classroom infrastructure or a better curriculum or assessment structure.
There is a notable difference between the enthusiasm that policy makers and teacher trainers show for ICT in education and its effective use in classrooms. Whenever I present the planning of our education programme, the part on computer hardware gets most attention and questions, the part on student-centred approaches, low-cost experiments or (printed) posters very much less so. Cuban uses a metaphor of hurricanes to describe educational reforms. The high waves at the surface are the grand policy statements by politicians. Underneath the surface the turbulent water is alike educational technologists who predict and analyse the policy effects. At the bottom of the sea however water flow is hardly affected by the hurricane. Similarly, teachers in classrooms soldier on, dealing with invisible barriers and ‘details’, that in practice make all the difference.
Such ‘details’ include poor uptime due to lack of maintenance, low technical skills, lack of and unstable power supply, causing technical defects to adapters and batteries and a school culture that considers ICT equipment as a ‘trophy’ to protect rather than as an instrument to use. Its use for learning is only as good as the teacher in the classroom. In Peru – as in Cambodia – with exceptional teachers, it becomes a useful tool, improving dialogic and problem-based learning. With an ordinary teacher, it is just a means of entertainment and reinforces the teacher-driven mode of instruction.
An interesting aspect of the OLPC discussion is the question how to measure its success. ‘Believers’ and ‘sceptics’ use a different measuring stick. OLPC implementation programs in Peru and Uruguay also expressed different objectives:
In assessing a program’s effectiveness, it’s important to distinguish the difference between outputs and outcomes as well ensure alignment with measurement and evaluation criteria. Thursday’s discussion pointed out Uruguay’s clear objective of social inclusion, which produced a near 100% primary school penetration rate through a national 1-to-1 program. The Uruguay assessment focused on access, use, and experience, reflecting a focus on social inclusion as an outcome. In the case of the assessment of Peru, math, language, and cognitive test results showed outputs, but no clear connection to Peru’s 2007 stated objectives which targeted pedagogical training and application. If objectives and outcomes are not clearly aligned with assessment criteria, can “effectiveness” be appropriately measured?
Cuban uses the ‘black box’ metaphor for classroom practice. Inputs (computers, new curricula, lab materials, pedagogical innovations) are introduced and outputs (learning outcomes, usage data) are collected, but with little information on what goes on in the classroom. Regular (and preferably unexpected) lesson observations and many interviews are probably the only way to get insight into what goes on in the ‘black box’. However, they don’t yield the ‘hard’ data that reporting with SMART indicators requires.
For me, the OLPC approach illustrates the failure in practice of the concept of ‘minimally invasive education‘, popularized by Sugatra Mitra’s ‘Hole in the Wall’ project in India. This approach claims that children’s ‘natural inquisitiveness’ combined with a computer is sufficient to get them learning. Audrey Watters hits the mark by stating that ‘there remains a strange tension between dropping in a Western technological “solution” and insisting doing so is “non-invasive”.
Anyway, the debate on the project is excellent and inspiring, perhaps more than the evaluation study itself. Programme leaders joined in the discussion. It illustrates that success and failure in such kind of development projects are up for debate. A debate that will inspire future project formulations. Feel free to add your thoughts and suggestions in the comments!