Key concepts and definitions
4.9 External validity
As mentioned above, one of the main reasons for choosing to conduct a quantitative study is that the interpretations from the data collected may be generalised from the sample to the population. However, for any quantitative study it is necessary to examine whether or not the study has been conducted according to certain principles which are required for generalisation to be possible. In more technical terms: it is necessary to examine the survey’s external validity. External validity questions a study’s generalisability, and examines to what extent, and within which population(s) and setting(s), the results of a study may be generalised (Trochim 2005).
Any discussion of external validity must investigate whether the study sample is in fact representative for the population. The foundations for a representative sample are laid through random selection, but even when the sample is in fact randomly selected, a large percentage of non-respondents may threaten its generalisability. As a consequence, there are two main issues which must be discussed for before it is sound to conclude that any results of the present study have external validity:
1) The significance of non-respondents
2) The inclusion of two non-randomly selected schools as substitutes for non-respondents
I will begin by addressing the first of these two issues before moving on to the second.
4.9.1 Non-respondents
There were three schools from the original random sample that did not wish to participate in the study. Of these, one was located in Hordaland, one in Oslo and one in Akershus. The problem with non-respondents is that it is always possible that they share certain traits which make them prone not to respond. For example, it has often been the case in Social Science research that respondents of lower economic status are less likely to respond to questionnaires sent in the mail than people of higher status (Johannessen, Tufte &
Kristoffersen 2005). The result is of course that researchers gain a limited perspective of the phenomena they aim to study. As a thought experiment, we can imagine that the non-responding schools chose not to participate because the teachers at these schools have little regard for the importance of the English programme subjects. It is possible that such attitudes would have affected the students at these schools, and that they would have reported answers which diverged from the rest of the respondents had they participated. It is also possible that if 1/3 of the original sample feels this way, then 1/3 of the population do too. If so, the fact that three schools did not wish to participate in the study may have resulted in erroneous data. This thought experiment illustrates how the nonparticipation of three schools may have had an effect on the external validity of the study results.
Fortunately, however, it does not appear that some schools declined to participate in my study due to a disregard for the English programme subjects. Instead, they explained their non-participation with reference to a general policy of not voluntarily taking part in any studies which would steal time away from teaching. Regarding the existence of three non-participating schools of the original random selection, I therefore conclude that there is nothing which indicates that these schools are in any way special or interesting.
Consequently, it does not seem likely that the study has missed out on vital information because of their failure to participate.
4.9.2 Non-randomly selected sample
As mentioned above, when three of the randomly selected schools declined to participate in this study, I decided it would be necessary to replace them in order to keep the number of respondents at an acceptable level. As time was of the essence, I approached some personal contacts within the Norwegian school system, instead of choosing random schools from the sampling frame. I recruited two substitute schools in this fashion. Because these schools
were not randomly selected, it was necessary to examine their impact on the results before it was possible to conclude anything regarding the external validity of the study. To do so, I needed to compare the responses given by respondents belonging to the seven randomly selected schools to the results provided by all respondents combined. By doing so, it would become apparent if the inclusion of two non-randomly selected schools had affected the results in any noticeable ways.
Fortunately, the results of this examination revealed that the differences between the randomly-selected schools and the non-randomly selected schools were minimal. Even more importantly, their impact on results was minimal. There are no indications that the two non-randomly selected schools differ much, or at all, from the rest of the schools in the sample, at least not in any way which exceeds the differences between the schools in the randomly selected sample segment. As a result, I have decided to treat all schools the same, and examined them as a unified sample.
4.9.3 Conclusion – external validity
I believe there is little reason to expect that the inclusion of two non-randomly selected schools in the sample of this study threatens the external validity of any results. The main reason for this is that, as seen, the answers provided by the non-randomly selected schools do not appear to affect the outcome of the study results in any major way. In addition, there is little reason to suspect that the three non-participating schools were in any way different from those who agreed to participate, and that their failure to contribute “hide” relevant information. Also, with the exception of the one non-participating school in Hordaland County, the geographical distribution is the same for both the original sample and the final sample. All in all, there are few reasons why the present sample should not be treated as if it was an entirely random sample.
4.10 Summing up
In this chapter I have presented the process of writing my master thesis with particular focus on challenges which appeared along the way. More specifically, Ch.4 Method, started with a presentation of how I came to choose the research question which is presently being studied.
Next, it addressed how I defined the study population and put together a sampling frame.
Furthermore, there was focus on the strengths and weaknesses of choosing to conduct a
quantitative study. I also described how the survey itself was constructed, and reflected on its strengths and weaknesses. Finally, much emphasis was placed on the sample selection process, and the fact that even though I will treat the sample as randomly selected, two of the schools included were in fact recruited through contacts rather than at random. Luckily, there is little which indicates that the two non-randomly selected schools are in any way different from the random sample. This is why I have decided to treat my sample as I would have had it been 100 % randomly selected.
In the next two chapters, I present the results of my study. Ch.5 Results A deals with students who have chosen to study English, while Ch.6 Results B deals with students who have not.