Constructing a survey

As mentioned in Ch.1 Introduction, I conducted a small-scale pilot study as part of a course paper prior to commencing work on my master thesis proper. In this pilot study students at two upper secondary schools in Oslo were asked to answer a questionnaire which was somewhat similar, but by no means identical, to the questionnaire used in the present study.

The pilot questionnaire was principally based on hunches about why English programme subjects are chosen/not chosen, and only marginally informed by theory and other studies.

Such an approach to questionnaire design is not advisable, and the development of this master thesis survey had to be more solidly founded on theory.

I decided motivation theory, particularly related to second language learning, would provide a sound and well-researched frame for my study. As has already been addressed in the previous chapter, it seemed beneficial to construct a questionnaire which would examine aspects from several different motivation theories, instead of settling for one theoretical perspective only. Few studies have been conducted of subject choice in Norwegian schools since the introduction of the LK06 curriculum, and I could therefore not rely on previous studies as sources of information as to which theories would be relevant and appropriate for the present study of subject choice. The solution was to construct a questionnaire which would cover a wide selection of possible explanations of subject choice. This questionnaire should facilitate an examination of relevant aspects pertaining to orientations, as formulated by Gardner and Lambert (1972), Clément and Kruidenier (1983), Dörnyei (1990), and Noels (2001), Expectancy-Value theory as formulated by Eccles et al. (1983), and Eccles and Wigfield (2002), and Self-determination theory as discussed by Deci and Ryan (1983). For the most part, the items which were included in the questionnaire were based on existing literature within the field of motivational research. In addition, some items were informed by previous studies of subject choice (see Ch.1 Introduction for details), while others were created specifically for the present Norwegian setting. By combining the instinctive hunches examined in the pilot project with the experience of several different and well-researched motivation studies, I believed I have contributed to increasing the construct validity of the present study.

Because I wanted to examine ideas from several different theories I knew my survey had to be quite extensive. However, Dörnyei (2007) warns that surveys should rarely take more than 30 minutes to complete, as respondents might lose interest or become irritated if they have to spend more time than this answering. In addition, teachers are constantly being asked to spend time on activities which have little or nothing to do with their students’

learning, and therefore it is quite possible that it would have been difficult to recruit participants if the study would take a long time to complete. Therefore, I concluded that asking to “steal” no more than 15 minutes of the students’ time would be an advantage when attempting to recruit participating schools.

However, using a survey which is short can also be negative because it makes it difficult to use multi-item scales. Multi-item scales refer to clusters of several differently worded questions which focus on the same variable. In ordinary multi-item scales the answer to all the multi-item questions will be summed up, resulting in a total score. The idea is that discrepancy in response given to questions which are intended to address the same variable – due to misunderstandings or misinterpretations – will be evened out through a summation of scores. This depends on there being more than just a couple of questions aimed at each target, and Dörnyei (2007) recommends an absolute minimum of four.

The flipside of multi-item scales is that some respondents may react negatively to being asked several questions about the same topic. Some may feel that it is a test of their honesty, or that the survey is more tiresome and tedious than it needs to be (Dörnyei 2007).

Furthermore, multi-item scaling needs multiple questions for each topic examined, which frequently leads to quite comprehensive questionnaire forms. However, as mentioned above, I wanted the questionnaire to be fairly comprehensive, but still take no more than 15 minutes to complete. In order to achieve this without exceeding the allotted time limit, I decided against constructing a survey in line with true multi-item scaling. Instead, I use some of its ideas in order to examine the harmony between various questions.

In my survey, there are generally two or three questions aimed at each of the elements examined, sometimes four, but never more. Instead of summing up the answers to these questions, I examined answers using simple correlation. There was for the most part a correlation of between r=.2 and r=.6, significant at a .01 level, between similar items, which indicates that students who agree to one question examining for example interest will most often agree to another question examining the same thing. Even though this is not a proper and fool-proof way of examining the validity of items, it provided some indication of whether or not the survey questions had been interpreted in the way they were intended. For an example, please consult Appendix 4.1, where I have given an overview of the correlations within three main categories of items related to why English is chosen.

The version of the survey which was used to collect data for this thesis ended up having 33 questions which all students, regardless of subject choice, were to answer. In addition, there were 20 questions particular to students who had studied one or more English subjects, and 21 for the students who had not. I piloted this survey at one upper secondary school in Oslo, and asked the teacher who helped me to notice how long students spent answering my questions. According to her, all respondents used less than 15 minutes. After

this concluding pilot I made some minor changes to the wording of three questions, and decided that the result was suitable for use.