The analysis of the Ardalan et al paper, that compares students’ responses to paper-based and online course evaluation surveys, for TMA03 made me look at a paper from Mantz Yorke (Yorke, 2009) that empirically analyses the effect of some design elements in student experience surveys. The paper is worthwhile alonefor its extensive literature overview of research findings and underlying psychological constructs that attempt to explain those findings.
Schematic overview of Yorke (2009) paper
In the empirical part of the paper the author looks at 4 research questions:
- Does the directionality of the presentation of a set of response options (‘strongly agree’ to ‘strongly disagree’, and vice versa) affect the responses?
- When there are negatively stated items, does the type of negativity affect the outcome?
- Does using solely positively stated items produce a different response pattern from a mixture of positively and negatively stated items?
- Does having negatively stated items in the early part of a questionnaire produce a different pattern of responses than when such items are left until later in the instrument?
Despite the lack of statistically significant findings the author writes:
‘Statistically non-significant findings seem often to be treated as if they were of no practical significance. The investigations reported in this article do, however, have a practical significance even though very little of statistical significance emerged’ (Yorke, 2009, p.734).
The nature of the reflection will depend on the context, such as the purpose (formative vs. summative) of the survey and the local culture (Berkvens, 2012). The author offers a rich overview of items that should be part of such a reflection and discusses explanatory frameworks from psychology. Unlike the Ardalan paper, the attempt to explain findings by referring to psychological theory moves the paper beyond mere correlations and creates causal and predictive value.
In a paper for the Learning Analytics Conference of 2012, Arnold and Pistilli explore the value of learning analytics in the Course Signals product, a pioneering learning analytics programme at Purdue University. The researchers used three years of data from a variety of modules. For some modules learning analytics was used to identify students ‘at risk of failing’, based on a proprietary algorithm that took into account course-related factors such as login data, but also prior study results and demographic factors. Students ‘at risk’ were confronted with a yellow or red traffic light on their LMS dashboard. Based on the information tutors could decide to contact the student by e-mail or phone. The researchers compared retention rates for cohorts of students who entered university from 2007 until 2009. They complemented this analysis with feedback from students and instructors.
Modules with use of CS showed increased retention rates – likely due to the use of CS. These courses also showed lower than average test results, possible a consequence of the higher retention. Student feedback indicated that 58% of students wanted to use CS in every course, not a totally convincing number.
The research paper generated following issues/ questions:
- The correlation doesn’t necessarily point to a causal link (although the relation seems quite intuitive)
- It’s unclear how courses were selected to be used with CS or not. Possibility of bias?
- The qualitative side of the research seems neglected. Interesting information such as the large group of students who are apparently not eager to use CS in every course is not further explored.
- The underlying algorithm is proprietary and is thus a black box for outsiders, which severely limits its applicability and relevance for others.
- It’s unclear with exactly what the use of CS is compared. If students in non-CS modules get little personal learner support, CS may look like a real improvement.
- The previous point relates with the need for clear articulation what the objective(s) of CS or learning analytics in general are. Including an analysis of tutor time saved or money saved through retention rates would have given a more honest and complete overview of the benefits that are likely perceived as important, instead of a rather naive focus on retention rates.
- It;s unclear if and how informed consent of students is obtained. Is it part of the ‘small print’ that comes with enrolment?
- How about false positives and negatives? Some students may get a continuous red light or face a bombardment of e-mails, if they belong to a demographic or socio-economic group ‘at risk’. Others may complain when they don’t receive any warnings despite having problems to stay in the course.
- The authors have been closely involved in the development of the learning analytics programme at Purdue University. This raises questions about objectivity and underlying motives of the paper.
Arnold, K.E. and Pistilli, M.D. (2012) ‘Course signals at Purdue: using learning analytics to increase student success’, In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge
, LAK ’12, New York, NY, USA, ACM, pp. 267–270, [online] Available from: http://doi.acm.org/10.1145/2330601.2330666
Week 12 in the H809 course and MOOCs – the official educational buzzword of 2012 – couldn’t remain absent. The focus in this course is not so much on what MOOCs are, their history and the different types with their various underlying pedagogies and ideologies. I blogged on MOOCs before, as a participant in LAK11, a connectivist MOOC on learning analytics. In H809 the focus lies on issues such as:
- What kind of information and research is available on MOOCs?
- What kind of MOOC research would be interesting to do?
- What are benefits and limitations of the type of information on MOOCs that is around?
- What is the educational impact (rather than the press impact) of MOOCs?
Much information on MOOCs consists of the so-called grey literature. Main information sources include:
- blogs from practitioners and academics, with an overrepresentation of academics from Athabasca Un. and the OU.
- blogs from participants in MOOCs, sharing their experiences
- articles in open academic journals such as IRRODL, EURODL, Open Praxis
- articles in more popular education magazines such as Inside Higher Education and The Chronicle of HE.
- articles in the general press such as The Economist and The New York Times
Some comments on these sources:
- The term ‘grey literature’ may sound a bit disparagingly. However, as Martin Weller writes, notions of scholarship and academic publishing are evolving. Blogs and open journals constitute alternative forms of scholarship with more interaction, less formality and shorter ‘turnaround’ times.
- Information and research on MOOCs is heavily Anglo-Saxon centred (or perhaps better Silicon Valley-centred?). I couldn’t hardly find any articles on MOOCs in Dutch, although that might not be so surprising. Although MOOCs (xMOOCs) are often touted as a ‘solution’ for developing countries, there are few perspectives from researchers from developing countries. As Mike Trucano writes on the EdTech blog from the World Bank:
“Public discussions around MOOCs have tended to represent viewpoints and interests of elite institutions in rich, industrialized countries (notably the United States) — with a presumption in many cases that such viewpoints and interests are shared by those in other places.”
- It’s interesting to see how many of the more general news sources seem to have ‘discovered’ MOOCs only after the Stanford AI course and the subsequent influx of venture capital in start-ups such as Coursera, Udacity and edX. The ‘original’ connectivist MOOCs, that have been around since 2008, let alone open universities are hardly mentioned in those overviews. A welcome exception is the Open Praxis paper from Peter and Deimann that discusses historical manifestations of openness such as the coffee houses in the 17th century.
- The advantage of this grey literature is that it fosters a tremendously rich discussion on the topic. Blog posts spark other blog posts and follow-up posts. Course reflections are online immediately after the course. Events such as a failing Coursera MOOC or an OU MOOC initiative get covered extensively from all angles. This kind of fertile academic discussion can hardly be imagined with the closed peer-review publication system.
- The flipside of this coin is that there are a lot of opinions around, a lot of thinly-disguised commercialism and a lot of plain factual mistakes (TED talks!). MOOCs may be heading for a ‘trough of disappointment’ in Gartner’s hype cycle. Rigorous research would still be valuable. For example, most research is descriptive rather than experimental and is based on ridiculously small samples collected in a short time. Interrater reliability may be a problem in much MOOC research . Longitudinal studies that investigate how conversations and interactions evolve over time are absent.
- Sir John Daniel’s report ‘Making Sense of MOOCs‘ offers a well-rounded and dispassionate overview of MOOCs until September 2012.
Interesting research questions for research on MOOCs could be:
- What constitutes success in a MOOC for various learners?
- How do learners interact in a MOOC? Are there different stages? Is there community or rather network formation? Do cMOOCs really operate according to connectivist principles?
- What are experiences from MOOC participants and perspectives of educational stakeholders (acreditation agencies, senior officials, university leaders) in developing countries?
- Why do people choose not to participate in a MOOC and still prefer expensive courses at brick-and-mortar institutions?
- What factors inhibit or enhance the learning experience within a MOOC?
- How to design activities within a MOCO that foster conversation without causing information overload?
- How do MOOCs affect hosting institutions (e.g. instructor credibility and reputation) and what power relations and decision mechanisms are at play (plenty of scope for an activity theoretical perspective here).
A few comments:
- High drop-out rates in MOOCs have caught a lot of attention. Opinions are divided whether this is a problem or not. As they are free, the barrier to sign up is much lower. Moreover, people may have various goals and may just be interested in a few parts of the MOOC.
- MOOCs (at least the cMOOCs) are by its nature decentralized, stimulating participants to create artefacts using their own tools and networks, rather than a central LMS. cMOOCs remain accessible online and lack the clear start and beginning of traditional courses. This complicates data collection and research.
- Although MOOCs are frequently heralded as a solution for higher education in developing countries, it would be interesting to read accounts from learners from developing countries for whom a MOOC actually was a serious alternative to formal education. The fact that MOOCs are not eligible for credits (at the hosting institution) plays a role, as well as cultural factors, such as a prevalent teacher-centred view on education in Asian countries.
Overview of posts on MOOCs from Stephen Downes: http://www.downes.ca/mooc_posts.htm
Overview of posts on MOOCs from George Siemens: https://www.diigo.com/user/gsiemens/mooc
OpenPraxis theme issue on Openness in HE: http://www.openpraxis.org/index.php/OpenPraxis/issue/view/2/showToc
IRRODL theme issue on Connectivism, and the design and delivery of social networked learning: http://www.irrodl.org/index.php/irrodl/issue/view/44
Armstrong, L. (2012) ‘Coursera and MITx – sustaining or disruptive? – Changing Higher Education’,
Peter, S. and Deimann, M. (2013) ‘On the role of openness in education: A historical reconstruction’, Open Praxis, 5(1), pp. 7–14.
Key paper in week 12 of H809 is a research paper from Davies and Graff (2005) investigating the relation between students’ activity in online course forums and their grades. It might be either due to the course team’s selection of papers or to our wider familiarity with methodological gaps in much educational research, but I found this paper to suffer from some rather obvious shortcomings.
- There is a problem with the operationalization of the concepts of participation and learning, or the construct validity. Participation has been quantified as the number of logins in the Blackboard LMS, and learning as the final grades. These are simplifications and the paper should at least discuss how these simplifications may distort results.
- There could well be co-variance between the factors. Both participation and learning may be influenced by third variables, such as prior knowledge, motivation, age… and a multivariate analysis might be more suitable to reveal these relations. There is no discussion in the paper about these underlying variables and possible co-variances.
- The question whether participation influences final grades may be irrelevant, as participation arguably has other beneficial effects for students beyond a possible effect on grades. Participation helps to foster a sense of community, may reduce feelings of isolation with some students and can promote ‘deeper’ learning. These perceived benefits are mentioned in the introduction of the paper, but not discussed in the conclusions.
- The study is based on a sample of 122 undergraduate students from a 1st year of a business degree. The sample size is quite small to get statistically significant results and is certainly too narrow to make sweeping conclusions about the relation between interaction and learning. One could argue what the objective is of a quantitative analysis on such a l
- The course context likely plays a strong role in the relation between interaction and learning. Variation between courses is higher than variation within a course, suggesting an important role for course design. Interaction in a course is not something that happens automatically, but it needs to be designed for, for example using a framework like Salmon’s e-tivity model. We don’t learn a lot about the context where the research took place. Did interaction took place through asynchronous or synchronous communication? Were there face-to-face interactions? Was the student cohort subdivided into smaller tutor groups? Lack of insight in the context limits the external validity of the research.
- I would argue that for this kind of research the analysis of outliers would be interesting (See Outliers from Malcolm Gladwell and Black Swans from Nassim Nicholas Taleb). The relation between online participation and course grades is not very surprising, but the correlation is far from perfect. Analysing learners who did interact a lot, but achieved poor grades and vice versa would yield insights in the circumstances when the relation is valid. This would result in more predictive knowledge at the student level about when non-participating students are at risk of failing. This relates to the next paper about the Course Signals project at Purdue University where learning analytics is used to devise a kind of early warning system for students. Interestingly, the (proprietary) algorithm uses variables such as residency, age and prior grades (together with participation, measured by logins in the course system) as predictors for identifying students at risk.
Two key terms in H809, originally introduced by Campbell and Stanley (1963) and often confused. Validity in itself is a contested term, with a variety of category schemes designed over the years. Below a scheme summarizing the two terms, based on references recommended in the course text.
Apart from focusing on validity, reliability and its sub-categories, the course texts suggests using a list of critical questions to evaluate research findings, such as:
- Does the study discuss how the findings are generalisable to other contexts?
- Does the study show correlations or causal relationships?
- Does the study use an underlying theoretical framework to predict and explain findings?
- How strong is the evidence? (in terms of statistical significance, triangulation of methods, sample size…)
- Are there alternative explanations?
Scheme summarizing validity and reliability, based on Trochim (2007)
The Hawthorne effect, the name derived from a series of studies in the 1920s at the Hawthorne Works manufacturing plants in the mid-western US. It’s often misinterpreted (‘mythical drift’) as a kind of scientific principle, describing the effect that the researcher has on the experiment, or the effect of the awareness by those being studied that they’re part of an experiment. In reality, the Hawthorne studies are useful to highlight some of the pitfalls of dealing with people (both the researcher as the research objects) in research.
- Anon (2009) ‘Questioning the Hawthorne effect: Light work’, The Economist, [online] Available from: http://www.economist.com/node/13788427 (Accessed 28 April 2013).
- Olson, Ryan, Hogan, Lindsey and Santos, Lindsey (2005) ‘Illuminating the History of Psychology: tips for teaching students about the Hawthorne studies’, Psychology Learning & Teaching, 5(2), p. 110.
The second paper in week 11 of H809 looks at the effects of the medium when soliciting course feedback from students. A switch from paper-based to web-based survey methods (2002-2003) provided a natural experiment setting for Ardalan and colleagues to compare the two modes for a variety of variables. As for the Richardson paper , we were asked to critically look at the methodology and issues such as validity and reliability. A lively (course-wide) forum helps to collect a variety of issues.
Schematic representation of Ardalan et al.(2007) paper
- The study aims at presenting a ‘definitive verdict’ to some of the conflicting issues surrounding paper-based and web-based surveys. The paper clearly favours statistically significant correlations as proof. However, despite the large sample, the research is based on courses in one North-American university (Old Dominion University, Virginia) during two consecutive academic years (2002-2003). The context of this university and academic years is not described in detail, limiting the applicability of the paper to other contexts. Generalisability could be enhanced by including more institutions over a longer period of time.
- The study succeeds in identifying some correlations, notably effects on the response rate and the nature of responses (less extreme). However, it doesn’t offer explanations for the differences. Changes in response rates could be due to a lack of access to computers by some students, they could be due to contextual factors (communication of the survey, available time, incentives, survey fatigue…), or they could be due to fundamental differences between the two survey modes . We don’t know. The study doesn’t offer an explanatory framework, sticking to what Christensen describes as the descriptive phase of educational research.
- It’s a pity that the study wasn’t complemented by interviews with students. This could have yielded interesting insights in perceived differences (response rates, nature) and similarities (quantity, quality).
- I found the paper extremely well-structured with a clear overview of literature, research hypotheses,
- The difference response rate may well have had an impact on the nature of the sample. The two samples may have been biased in terms of gender, age, location, socio-economic status (access to web-connected computer). Perceived differences between the modes may have been due to sample differences.
- I’m not sure whether the research question is very relevant. Potential cost savings for institutions from switching to web-based surveys are huge, making that institutions will use online surveys anyway.
Even a medium-size institution with a large number of surveys to conduct realises huge cost savings by converting its paper-based surveys to the web-based method. With the infrastructure for online registration, web-based courses and interactive media becoming ubiquitous in higher education, the marginal cost savings above the sunk costs of existing infrastructure are even more significant. (Ardalan et al., 2007, p.1087)
Lower response rates with web-based surveys can be dealt with by increasing the sample size. Rather than comparing paper-based and web-based surveys (a deal that is done anyway), it would be more interesting to analyze whether web-based surveys manage to capture a truthful image of the quality of a course as perceived by all students and what are influencing factors and circumstances.
The paper from Richardson (2012) investigates whether the persistent attainment gap in higher education is affected by the tuition mode. Arguments can be made that online tuition both widens and narrows the gap. The paper looks to answer 1/ whether ethicity affects the choice for face-to-face vs. online tuition, and 2/ whether ethnicity patterns were different in both tuition modes.
I’ve summarized the main elements of the paper in the scheme below. I wasn’t impressed with the findings. The main limitations seemed to be the narrow sample and the sole focus on ethnicity which, in my opinion, is not an explaining variable for student performance, but rather a proxy for other socio-economic and cultural variables. These should be explored in more detail in order to gain a better understanding of this attainment gap.
Schematic representation of Richardson (2012) paper
– sample limited to 1 university and 2 courses
– two modules yield different outcomes (possibly due to variance in online tuition quality)
– self-selected sample: are characteristics of learners choosing online/ f2f tuition mode identical?
– ethnicity proxy variable for other factors affecting attainment (internet access, job situation, family status, geographical factors)
– little insight in reasons why learners choose particular mode of tuition.
– unclear how learners themselves assess the quality of tuition.
Richardson, J.T.E. (2012) ‘Face-to-face versus online tuition: Preference, performance and pass rates in white and ethnic minority students’, British Journal of Educational Technology, 43(1), pp. 17–27.
Only a few papers per module truly change the way you look at a topic, through the clarity and profoundness they build their thesis. The second reading in week 8, from Crook and Dymott (2005) is such a paper. It looks at the effect of ICT on writing in an undergraduate course. By teasing out the question and showing that writing as an activity comprises much more than entering letters on a screen, it explores a wide range of effects.
Consider the text you are currently reading. Composing the preceding paragraph certainly involved keyboard tapping and screen staring: activities that perhaps could be photographed as ‘me writing’. However, much more was involved than those simple behaviours. There was the business of distributing attention. This applies to the screen, the keyboard, and a set of paper notes. But also – because of a social interruption – it applies to the screen and other forms of event on the periphery. Writing is organized (it is located and scheduled) to be in a useful harmony with this periphery: sometimes exploiting it, sometimes needing to be insulated from it. (Crook and Dymott, p.96-97)
The paper explores the difference between a traditional psychological approach and one inspired by socio-cultural theory. Crook and Dymott discuss some limitations (in their view) of traditional psychology.
In traditional psychology, with the individual as unit of analysis, the question would relate to:
- isolation of design features (‘isolating the skill from its users’)
- set up experimental design: ‘approaching ICT as a convenient cluster of independent variables’
- focus on outcomes (‘narrow behavioural repertoire‘)
- little concern for processes
- separation of literacy skills (writing, reading, searching and selecting…)
In socio-cultural psychology, writing is considered as a cultural practice with ICT and writing mutually constitutive.
- writing is considered as a rich, complex system of activities, incl. reading, searching, note-taking, discussing, reflecting…
- writing is individual acting with mediational means (echoing Vygotsky)
- writing inherently connected with technology (pens ,paper, computer…)
- focus on revealing structures underlying practice
- focus of inquiry is on the processes of learning and on meaning making in social settings.
- research in authentic processes rather than designs: ‘Most seem to concern themselves with the control of so many variables that the resulting experimental task bears little resemblance to the activities most of us routinely perform as ‘reading’. (p.100)
- methods: observation and analysis (diaries, interviews, activity logs), complemented by controlled forms of study
- simple generalisations about singular effects are inappropriate, individually determined
Crook and Dymott compare the introduction of ICT with the introduction of guns. Guns don’t ‘strengthen’ or ‘amplify’ the killing of animals, rather they change the process and relations in the community.
Case study: writing by undergraduate students
Five aspects of the writing process are discussed and how ICT affects them.
1/ text on screen
- Numerous differences between paper and screen
- Does a computer screen facilitate or inhibit writing (issues of legibility, global structure text, skim reading)?
- Depends on individual factors (prior experiences…)
- Affordances: derive not fromtechnology per se, but from its relation with person (Different from original meaning of affordances, coined by Gibson)
2/ text on network
- interactive dimension of writing (management of document access; attentional commitments to sources)
- writing more in 1 location: disturbance of socially-distributed nature of writing. (differences between writings by students with and without network access)
- multi-tasking: ICT creates additional layer of alternative possibilities
3/ text as electronic traffic
- text more easily sent and received
- sharing of notes and work (individual differences)
4/ text and the website
- changes in notions of audience: sense of genuinely writing for others is important to cultivate
- make learner activity within classroom visible
- vicarious learning from access to work of peers
5/ dialogue around text
- proceduralisation of feedback (e.g. use of standard forms)
- create a sense that a social exchange has been reified
- mechanisation of communication
- marginal scribbles and summary comment: convey a sense of authentic dialogue with the reader
Crook, C. and Dymott, R. (2005) ‘ICT and the literacy practices of student writing’, In Monteith, M. (ed.), Teaching Secondary School Literacies with ICT, Maidenhead, Open University Press., pp. 97–113.
Conole et al. (2004) advocate the use of toolboxes as ways to bridge theory and practice. Practitioners don’t have time to wade through wads of theoretical papers. As a result many designs are based on ‘commonsense’ rather than theoretically informed. The authors argue that theory-informed designs would improve quality and that toolkits are the ideal instrument to realize this:
They distinguish toolkits from wizards (which are black boxes, hiding the underlying decision process) and conceptual frameworks (which offer little practical use).
Some characteristics and key terms on toolkits in the article:
- for non-expert users to engage with theories
- elicit assumptions and theories
- decision-making systems
- reflect beliefs and assumptions of creator(s)
- guiding framework
- offer flexibility for local context
- informed decisions
- offer common language
- provide examples (if linked database)
- promote reflective practice
The toolkit presented in the paper is represented by a model
Learning activities such as brainstorming or presentation of materials can be mapped with the model, prompting reflection on the overall pedagogical balance and the types of learning supported.
The paper contains a welcome synthesis of learning theories. I’m less convinced about the practical value of the toolkit. Publishing the paper in a closed-access journal is not likely to contribute to its adoption by practitioners.
Communities of Practice is one of the most used concepts in educational research these days. Wenger (1998) has provided a theoretical basis for the concept, although his definition is quite fluid and difficult to grasp (Johnson, 2001). Preece (2000) has developed an operationalisation of the concept, centred around concepts of usability and sociability. These relate to the duality, developed by Wenger, between design and emergence.
Jones and Preece (2006) distinguish between Communities of Interest (COI) and Communities of Practice (COP). The latter, described by Wenger (1998), are reserved for communities in professional contexts. COI refer to the more organic, loosely structured communities that centre around people’s interests. Garrison has coined the term Community of Inquiry, focusing on groups in educational settings. There seems to be a rich amount of literature on these Communities of Inquiry.
Preece (2000) uses a sociability and usability framework to analyse the success of COI and COP. Usability is related to user-friendliness and consists of guidelines for the design of online spaces. Criteria for sociability centre around the 3 P’s of people, purpose and policies.
Sociability framework (Preece, 2000)
- requires ‘nurturing’ in young communities
- ‘lurkers’ routinely comprise at least 50% of participants
- Empathy and trust
- empathy: ability to understand others and react compassionately
- trust: expectations of positive interactions
- Clear leadership and commitment
- Supported by research from Wenger et al. (2011): “what makes a difference is not the quantity of users, but the passion and commitment with which a subset of users provide leadership, example and high quality content”
- Common ground
- corresponds with ‘mutual understanding’ (Wenger, 1998) , sense of unity, a common vision & values
- clarity of common purpose for participants (* I’m not convinced all participants need to have a common purpose)
- related to motivation
- Incentives for collaboration (vs. competition)
- can be realized through formal rules or through self-governance/ cultural norms
- related to amount of social pressure and presence of leadership
- Social presence
- described as sense people have online of others being present
- can be generated by short response time, not necessarily by many postings
- is strongly positively related with etiquette
- COI/COP need time to form and grow, in order to develop, in Wenger’s (1998) terms, ‘mutual understanding’, ‘common language’ and ‘reified artefacts’
Comparing these criteria with Stephen Downes’ description of characteristics of successful networks, highlights some of the differences between communities and networks:
- degree to which a network and its members can act independently
- not a criterion for a community, rather are coherence and a sense of belonging to group (identification)
- degree to which various backgrounds and opinions are represented in the network
- communities require a mutual understanding and shared repertoire.
- degree to which the community is open to new members
- * although not mentioned, I believe this was a major weakness of the COP of physics teachers in the Jones and Preece (2006) study.
Preece, J. (2000) Online Communities: Designing Usability, Supporting Sociability, John Wiley & Sons.
Jones, A. and Preece, J. (2006) ‘Online communities for teachers and lifelong learners: a framework for comparing similarities and identifying differences in communities of practice and communities of interest’, International Journal of Learning Technology, 2(2), pp. 112–137.
Wenger, E. (1998) Communities of Practice: Learning, Meaning, and Identity, Cambridge University Press.
Wenger, E., Trayner, B. and De Laat, M. (2011) Promoting and assessing value creation in communities and networks: a conceptual framework
, Ruud de Moor Centrum, Open University of the Netherlands, Available online