The analysis of the Ardalan et al paper, that compares students’ responses to paper-based and online course evaluation surveys, for TMA03 made me look at a paper from Mantz Yorke (Yorke, 2009) that empirically analyses the effect of some design elements in student experience surveys. The paper is worthwhile alonefor its extensive literature overview of research findings and underlying psychological constructs that attempt to explain those findings.
Schematic overview of Yorke (2009) paper
In the empirical part of the paper the author looks at 4 research questions:
- Does the directionality of the presentation of a set of response options (‘strongly agree’ to ‘strongly disagree’, and vice versa) affect the responses?
- When there are negatively stated items, does the type of negativity affect the outcome?
- Does using solely positively stated items produce a different response pattern from a mixture of positively and negatively stated items?
- Does having negatively stated items in the early part of a questionnaire produce a different pattern of responses than when such items are left until later in the instrument?
Despite the lack of statistically significant findings the author writes:
‘Statistically non-significant findings seem often to be treated as if they were of no practical significance. The investigations reported in this article do, however, have a practical significance even though very little of statistical significance emerged’ (Yorke, 2009, p.734).
The nature of the reflection will depend on the context, such as the purpose (formative vs. summative) of the survey and the local culture (Berkvens, 2012). The author offers a rich overview of items that should be part of such a reflection and discusses explanatory frameworks from psychology. Unlike the Ardalan paper, the attempt to explain findings by referring to psychological theory moves the paper beyond mere correlations and creates causal and predictive value.
Learning Matters 