Computers in Schools: Why Governments Should Do Their Homework

Time and time again, governments and NGOs herald the purchase of ICT as a panacea for improving the quality of education. The recent plans of Gauteng in South Africa are a good example. This study from the Inter-American Development Bank (IDB) provides an useful summary of the research done on the impact of ICT in primary level classrooms.  Latin America and the IDB have been at the forefront om some high-profile “One Laptop per Child” projects such as the Plan Ceibal (Uruguay), Enlaces (Chile) and the OLPC Programme in Peru, on which I blogged before.

Some extracts:

The evidence so far is quite persuasive that programs that overlook teacher training and the development of software may yield low returns.

One promising avenue lies in the use of ICT to realise productivity gains in school management:

The collection, transmission, and analysis of data on enrollment, absenteeism, test scores, and infrastructure can help principals spot a problem in a given classroom, administrators spot an exemplary school, and policymakers track the performance of the educational system and the resources available. However, the gains in productivity seen in the business sector are rarely seen in the educational system, some have argued, because most education managers are not knowledgeable in the use of information management tools.

Studies that measured the impact of ICT, both of the access to computers and the use of computers, found more often than not no significant impact on learning outcomes – an overview is included in the report.  The authors note that it’s not sufficient for ICT investments to produce a positive impact, they should produce a positive impact compared to traditional instruction and, even better, to similar investments in other areas such as teacher training, smaller classes or libraries.

All other things being equal, the impact of ICT investments will be higher when the quality of teaching is low, as the potential for learning gains is higher. This underlines the risk of extrapolating findings from developed to developing country context.

Some recommendations from the report:

  • Given the high investments, the low number of decent impact studies is surprising.  The impact of ICT investments heavily depends on the context and on the implementation.  As such, results from impact studies cannot be generalized over different programmes.  Start on a limited scale and build impact evaluation into the programme design is important.
  • Important to keep the Total Cost of Operation (TOC) of ICT investment into account rather than the purchase price. This includes maintenance, training, connectivity and electricity costs.  Recurrent costs typically take up about 40-50% of the initial investment (in Latin America).  These are permanent costs, which imply savings elsewhere in the education system or an overall increase in expenditure. A large share of rural schools, high electricity and connectivity costs and high wages (as in South Africa) thus increase the share of recurrent costs.
  • Most successful ICT project implementation focus on honing ICT skills of learners and pursuing Computer-Aided Instruction (CAI), for example for maths.

Why Gauteng’s Plan for Introducing Smartboards in Every Classroom Is Not a Good Idea

monkeysLast week the MEC for Education in Gauteng announced plans for replacing all blackboards by whiteboards.  This comes after earlier plans to distribute tablets and connect schools to the internet.

There is hardly any evidence for a positive relation between the amount of ICT in a classroom and learning outcomes. In some cases, this might be due to a mismatch with traditional assessment practices and curricula.  Some highly performing education systems are famously ICT-absent such as Finland.  Large-scale projects such as One-Laptop-per-Child didn’t find such a positive relationship (see also my blog post on IDB study of OLPC Peru) and are now largely discredited.  At the very least, learners should get a personal device and teachers should be extensively trained and guided during implementation.  They should get time to explore and make mistakes. As Michael Trucano writes: personal laptops does not yet make personalized learning.  Systems with a few tablets or computers per class have a dismal track record. Apart from that, tablets are arguably the worst possible choice of device, as they’re designed for consuming content, much more than for creating content. Finally, curricula and assessment practices are often not adapted to the different kind of education that ICT enables, so that teachers will rather fit computers into their traditional ways of teaching and learning.

South Africa spends 6,2% of its GDP on education (2013 data), about the same as Belgium. However, about 90% are personnel costs, leaving schools with little space for investing in infrastructure. ICT infrastructure is expensive, in particular when the Total Cost of Operation (TOC) is taken into account (which is rarely done), which includes costs for bandwidth, electricity, insurance, security and maintenance.  Often the TOC is 10 times the initial purchase cost of the equipment.  It’s hard to see how South African schools are going to finance this.

The lack of research evidence stands in stark contrast with the discourse of policy makers.  This is easy to understand.  ICT holds the promise of ‘fixing’ a ‘broken’ education system. Policy makers are seen to take ‘decisive action’ and to be ‘modern’, ‘innovative’, finding ’21st century solutions’ for current-day ‘digital natives’.  This optimism is based on the potential of ICT in education, rather than its actual track record.  What often happens is that policy makers, assisted’ by vendors, have sought to simply transfer ICT-related models and practices from classrooms in industrialized countries to less developed education systems in other parts of the world, assuming reliable electricity and connectivity, well-trained teachers, sufficient available time-on-task, highly literate students, space to implement student-centric pedagogies, relevant content, etc. that do not correspond with local realities.

South Africa consistently ranks near the bottom of international rankings of learning outcomes such as those from SACMEQ, TIMMS and WEF. Massive investments in ICT hardly seems like the most urgent priority in a system with massive challenges in teacher competence, basic infrastructure such as sanitation.  Rolling back-outs, called load shedding in South Africa, raise the question how all these computers will be charged.

Additionally, dotted with one of the most unequal countries in the world, it’s hard to see how investing in ICT will bridge the digital divide in education.  Despite rhetoric that argues the contrary, introduction of ICT in schools often exacerbate various entrenched inequities in education systems (urban-rural, rich-poor, boy-girl, linguistic and cultural divides, special needs students ). Mitigating these inequities is possible, but requires explicit attention

So, is there no role for ICT at all in South African education?  The following areas seem more promising to me than equipping each learner with a tablet and each teacher with a smartboard:

  1. ICT’s main added value initially lies in school and class management. Helping educators to keep electronic learner records, lesson plans, test results etc. can greatly improve efficiency and monitoring quality.
  2. Helping teachers to become information literate, including basic online search skills, collaborative editing and knowledge management greatly improves their access to knowledge and resources such as OERs.
  3. Simulations and animations have proven to be helpful in science lessons, where many processes are too slow (geological processes), too fast (falling objects), too big (planetary movements) or too small (atom interactions) to demonstrate.

Effectiveness of ICT in the classroom: Findings from the IDB study on OLPC Peru

Credit: jdebner

The Edutech Debate and the World Bank Blog from Michael Trucano regularly provide excellent background reading on the effectiveness on ICT in education (‘computers in the classroom’).  Discussion surged again with the publication of an Inter-American Development Bank (IDB) evaluation of the One Laptop Per Child (OLPC) project in Peru, which sparked a lengthy but worthwhile discussion on the EduTech website.

The IDB study on the OLPC programme in Peru did not find any tangible effect of the programme on students’ test results (national assessments in Math and language test results) 15 months after implementation (Link to Economist article on the study).

The IDB applied a randomized control experiment by sampling five students per grade per school out of 320 schools (2/3 of which were in a treatment receiving the intervention) at intervals of 3 and 15 months.  The recently published assessment brief covered the 15 month data.

Some people were quick to point to the impact of the programme beyond the immediate effect on test results.  Nicholas Negroponte, founder and chairman of the OLPC non-profit stressed that the purpose of OLPC was not to improve classroom learning only, but learning in the child’s whole life.  The computers arguably improved reading comprehension, parent involvement, critical and creative thinking, initiative and discovery. Also the OLPC Peru team defends the project and points out that attitudes and expectations of students, parents and teachers have changed (within 15 months of implementation of the project).  According to project leader Oscar Bacerro (reply to Economist and article in EduTech Debate) the Peru programme the educational system in such poor shape that improving the quality of its teachers would require 10-15 years. He focuses on the lack of a fertile environment in the Peruvian educational system to introduce the programme. 

“It was clear to us the main challenge for our project would not be “teacher training” on how to use computers in the classroom because most of our teachers needed exceedingly much more than ICT literacy courses.”

He argues that many teachers lack basic numeracy and literacy skills. Notwithstanding the fact that programme design included 40 hours of training for teachers, this was clearly insufficient for the majority of teachers.  The IDB analysis showed that almost half the students were prohibited from taking the XOs home and that half of all teachers didn’t use the devices in the classroom. Changes in cognitive abilities take many years to materialize.  Is this a case of too much emphasis on the infrastructure and not enough attention on technical and pedagogical support?

The OLPC project was not implemented in isolation.  There were several interventions aimed at improving the pedagogical and institutional framework of the schools.  A large in-service training programme was included in the programme design.  The OLPC part, of course, gets all the media attention.  I’m not sure what percentage of the budget went to infrastructure and what part to capacity development of teachers and educational stakeholders. In the VVOB programme in Cambodia the share of infrastructure is less than 20% which might even be too high.

Stanford professor Larry Cuban has been a famous sceptic of technology in the classroom since many years.  Cuban’s core point is that school improvement is hard and at its core is not about technology.  In fact, it’s the organization’s skill at defining a shared vision, communicating, collaborating, evaluating, changing, etc. that is the driver of effective outcomes. 

I do not doubt the valuable role ICT can play in education. I fully believe that technology can improve access to learning and its quality. However, I also believe that ICT in education often is about the least cost efficient way to improve education – with some notable exceptions.  Toyama refers to the difference between the purchase cost of a laptop and the total operation cost (TOC), which includes maintenance, electricity, software and connectivity.  The TOC is usually 5 – 10 times higher than the purchase cost, meaning that a 300 USD computer represents an investment of approx. 300 USD per year (assuming a generous 5 year computer lifetime).  This is quite a lot, taking into account that the Peruvian government spends on average 686 USD per child in primary education and 782 USD in secondary education – in Cambodia it is 54 USD/ child.  The question is whether this amount could not be better spent on improvements in teacher education, classroom infrastructure or a better curriculum or assessment structure.

There is a notable difference between the enthusiasm that policy makers and teacher trainers show for ICT in education and its effective use in classrooms.  Whenever I present the planning of our education programme, the part on computer hardware gets most attention and questions, the part on student-centred approaches, low-cost experiments or (printed) posters very much less so. Cuban uses a metaphor of hurricanes to describe educational reforms.  The high waves at the surface are the grand policy statements by politicians.  Underneath the surface the turbulent water is alike educational technologists who predict and analyse the policy effects.  At the bottom of the sea however water flow is hardly affected by the hurricane.  Similarly, teachers in classrooms soldier on, dealing with invisible barriers and ‘details’, that in practice make all the difference.

Such ‘details’ include poor uptime due to lack of maintenance, low technical skills, lack of and unstable power supply, causing technical defects to adapters and batteries and a school culture that considers ICT equipment as a ‘trophy’ to protect rather than as an instrument to use.  Its use for learning is only as good as the teacher in the classroom. In Peru – as in Cambodia – with exceptional teachers, it becomes a useful tool, improving dialogic and problem-based learning. With an ordinary teacher, it is just a means of entertainment and reinforces the teacher-driven mode of instruction.

An interesting aspect of the OLPC discussion is the question how to measure its success.  ‘Believers’ and ‘sceptics’ use a different measuring stick.  OLPC implementation programs in Peru and Uruguay also expressed different objectives:

In assessing a program’s effectiveness, it’s important to distinguish the difference between outputs and outcomes as well ensure alignment with measurement and evaluation criteria. Thursday’s discussion pointed out Uruguay’s clear objective of social inclusion, which produced a near 100% primary school penetration rate through a national 1-to-1 program. The Uruguay assessment focused on access, use, and experience, reflecting a focus on social inclusion as an outcome. In the case of the assessment of Peru, math, language, and cognitive test results showed outputs, but no clear connection to Peru’s 2007 stated objectives which targeted pedagogical training and application. If objectives and outcomes are not clearly aligned with assessment criteria, can “effectiveness” be appropriately measured?

Cuban uses the ‘black box’ metaphor for classroom practice.  Inputs (computers, new curricula, lab materials, pedagogical innovations) are introduced and outputs (learning outcomes, usage data) are collected, but with little information on what goes on in the classroom.  Regular (and preferably unexpected) lesson observations and many interviews are probably the only way to get insight into what goes on in the ‘black box’.    However, they don’t yield the ‘hard’ data that reporting with SMART indicators requires.

For me, the OLPC approach illustrates the failure in practice of the concept of ‘minimally invasive education‘, popularized by Sugatra Mitra’s ‘Hole in the Wall’ project in India.  This approach claims that children’s ‘natural inquisitiveness’ combined with a computer is sufficient to get them learning. Audrey Watters hits the mark by stating that ‘there remains a strange tension between dropping in a Western technological “solution” and insisting doing so is “non-invasive”.

Anyway, the debate on the project is excellent and inspiring, perhaps more than the evaluation study itself.  Programme leaders joined in the discussion.  It illustrates that success and failure in such kind of development projects are up for debate.  A debate that will inspire future project formulations.  Feel free to add your thoughts and suggestions in the comments!