#H809 Methodological Reflections on the Hiltz and Meinke Paper

The first paper in H809 is an oldie, a paper from Starr Roxanne Hiltz and Robert Meinke published in 1989.  The paper aims at comparing the learning outcomes in a few courses between online and face-to-face delivery.

Research Questions & Design
The article seeks to find out whether a virtual course implementation (VC) produces different learning outcomes than a traditional face-to-face (F2F) approach. Secondly, it looks to determine variables (student, instructor and course characteristics) associated with these outcomes.  The research uses quantitative research methods, using pre- and post-course survey data.  It complements these with evaluation reports from the course instructors.
The research aims at relating the mode of delivery to learning outcomes, measured by data such as SAT courses.  It takes a behavioural view on learning.  Alternatively, the research could have focused on the degree of understanding, the development of ‘soft skills’ 
Limitations of the research:
1. Distribution of students in groups (VC vs F2F) was done through self-selection (quasi-experimental approach).  Student characteristics may thus not be similar.  Perhaps, more disciplined or motivated students chose to take the VC approach.
2. The use of self-reporting pre- and post-course surveys may be prone to response bias. Responses may have been skewed by a desire to please the researchers. As students were asked to compare VC experiences with previous F2F experiences, they needed to rely on (distorting) memory. 
3. The scope of the research was limited to two institutions and a small student population.  Not surprisingly, few results were statistically significant: “In many cases, results of quantitative analysis are inconclusive in determining which was better, the VC approach or the F2F approach.  The overall answer is: It depends.”  Setting up methodologically sound quantitative research designs in a ‘real’ educational setting is challenging, as there are so many environmental variables that may influence the outcomes and which, in an ideal setting, should be kept constant in order to have conclusive results for the dependent variable.
4. The researchers mention implementation problems, such as resistance by faculty members.  Unfortunately, they don’t elaborate on this.
5. The same teacher, text and other printed materials were used in both modes.  This seems like an objective way to compare two modes, but it may not be. The teacher may have been less familiar with online delivery or failed to adapt his/her mode of instruction.  Texts and other printed materials may be suitable for F2F delivery, but online delivery calls for different course designs (See the work of Mayer and Clark).  For example, online delivery requires short chunks of text for online reading, proximity of a graph with the explanation of this graph and removal of redundancies in information.
6. The research focuses on the comparison of delivery modes (VC vs. F2F).  However, in their discussion on collaborative learning, the authors seem to suggest that it is mainly the selection of instructional strategies that counts, in particular the inclusion of collaborative learning activities like seminar-style presentations and discussions. 
The self-selecting of the samples is a weakness in the study.  Random selection would arguably provide a better basis to compare the learning outcomes of two delivery modes.  However, assigning students to a delivery mode, which you may suspect will put them to a disadvantage  and for which they have paid good money, raises ethical questions.  Providing the courses free of charge for students willing to take part of the study could be an option, although this may in turn affect the research.  Students may behave differently in a course for which they paid.
The study found little evidence of statistically significant correlations in learning outcomes between the two modes of delivery.  The pre- and post-test surveys did show some significant correlations in subjective assessments such as interest. Correlations were in both directions. For a mathematics course the online course generated higher interest course, whereas for an introductory sociology course, the result was opposite.  The authors suggest that this may be related to the fact that the sociology cohort was an academically weak group, as illustrated by their SAT scores.  
What counts as evidence in the paper?
The researchers look for statistically significant correlations.  I believe such a correlation gives more support for a claim, by indicating its strength and reliability. However, the claim is limited to the particular circumstances in which the research took place (characteristics of students, teachers, institutions, courses…) and cannot be extended to other circumstances without insight in the nature of the circumstances and their causality with the learning outcomes. In what circumstances do students achieve better learning outcomes in an online course?  For what types of courses does online learning offer a better learning experience?  The authors do discuss these circumstances, but base themselves mainly on their personal experiences as instructors rather than statistical tests.
A next step in the research could be to look for anomalies in the data.  Students, courses and implementation strategies that fail the hypotheses made.  For example the hypothesis that online learning is beneficial for more mature learners. Or the hypothesis that online learning is less suitable for wide, introductory courses that touch upon many topics.  
The research could form input to meta-analysis, which could compare the claims with other studies and try to distil findings based on a more diverse set of circumstances.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s