#H809 Issues with Student Experience Surveys

The analysis of the Ardalan et al paper, that compares students’ responses to paper-based and online course evaluation surveys, for TMA03 made me look at a paper from Mantz Yorke (Yorke, 2009) that empirically analyses the effect of some design elements in student experience surveys.  The paper is worthwhile alonefor its extensive literature overview of research findings and underlying psychological constructs that attempt to explain those findings.

Schematic overview of Yorke (2009) paper

Schematic overview of Yorke (2009) paper

In the empirical part of the paper the author looks at 4 research questions:

  1. Does the directionality of the presentation of a set of response options (‘strongly agree’ to ‘strongly disagree’, and vice versa) affect the responses?
  2. When there are negatively stated items, does the type of negativity affect the outcome?
  3. Does using solely positively stated items produce a different response pattern from a mixture of positively and negatively stated items?
  4. Does having negatively stated items in the early part of a questionnaire produce a different pattern of responses than when such items are left until later in the instrument?

Despite the lack of statistically significant findings the author writes:

‘Statistically non-significant findings seem often to be treated as if they were of no practical significance. The investigations reported in this article do, however, have a practical significance even though very little of statistical significance emerged’ (Yorke, 2009, p.734).

The nature of the reflection will depend on the context, such as the purpose (formative vs. summative) of the survey and the local culture (Berkvens, 2012).  The author offers a rich overview of items that should be part of such a reflection and discusses explanatory frameworks from psychology.  Unlike the Ardalan paper, the attempt to explain findings by referring to psychological theory moves the paper beyond mere correlations and creates  causal and predictive value.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s