This study will continue to explore decision-making activity surrounding the initial phases of evaluations of postgraduate programmes for school leadership development. Further delimited, this applies to understanding the subunit decision making process that results from the demand to evaluate and leading to the design of the subsequent evaluation implemented to meet this demand8. The overall focus of this study is related to the question:
What influences the decision of how postgraduate programmes for school leadership are evaluated?
It is recognised that this is a complex area, influenced by many different factors and variables. Therefore attention is delimited to 3 important and interlinked sub-questions related to their decision making about evaluations:
o What pressures and demands do subunits face?
o What design frameworks are available to them?
o What decision processes take place within subunits about the choice of evaluation model?
As has been stated, in order to answer this overall question it will be required to develop an understanding of the basic purposes and rationale of evaluation, as well as intentions for future utilization of findings and existing knowledge of factors thought to influence this process. When observing the response to this,
7 My translation from Danish
8 Based on decision theory outlined above (Allison & Zelikow, 1999; Peterson, 1976; Pfeffer, 1981b; Thompson, 2003) (Dahler-Larsen, 2006a) (Guskey, 2000), (March & Heath, 1994)
13
an interpretive design combined with a pragmatic approach is required, investigating members‘ understanding of their decision making with regard to the process of evaluation. In doing this understanding of the relationship to programme goals, content and underlying rationale will support analysis. This is outlined in the section below.
1.6 Methodology
In this research, analysis is made of the decision-making process through which designs for programme evaluation are adopted within subunits offering postgraduate programmes for school leadership. The unit of analysis is the organisational decision making process. The unit of observation will be the individual actors as members of subunits involved in the decision making process. With a lack of research in this area (Holton III, 2005) a pragmatic framework is constructed. At the same time it is recognised that theory has addressed this topic earlier even if has not been applied fully into the field.
Therefore an a priori theoretical and analytical framework is applied in the study. In this case it is an alternate templates strategy building on process rather than variance research (Langley, 1999). An alternate templates strategy involves analysing a process from a number of different perspectives and can involve both deductive and inductive approaches (Ibid.). The alternate templates strategy, is based on analysing and interpreting the same events through
―different but internally coherent sets of a priori theoretical premises‖, which are then assessed to the extent ―to which each theoretical template contributes to a satisfactory explanation‖ (Langley, 1999: 698). Each alone will, however, be insufficient despite its relevance. The explanatory power and accuracy of the models chosen here are considered to be increased when they are applied in tandem. According to Langley, this application of the different, but complimentary models can lead to data interpretation that may reveal
―contributions and gaps in each‖. Langley sees this strategy as similar to Allison‘s multi model approach and drawing also on Weick, Langley describes it as a process of sensemaking. This has particular relevance for this study.
Weick (1976) recognised that critical analysis is required of language and communication that facilitates the decision process, and in order to do this different theoretical perspectives should be held. Such research opens for a combined strategy of deductive use of theory and inductive use of data (Langley, 1999), which appears similar to Ragin‘s retroduction (1994), as well as the interactive research process described by Maxwell (1996). Such a strategy also appears close to that applied by Peterson (1976), which led to his nuanced view of Allison‘s third model. Instead of attempting to generalise, the intention is to develop propositions and limited theory by―[r]efining partial paradigms, and specifying the classes of actions for which they are relevant‖, (Allison, 1971 in Langley, 1999: 699). This method also has a similar rationale to that of
14
research revealing toxic decisions by Maitlis and Ozcelik (2004)9. Langley notes, however, that this approach can provide difficulties when attempting to combine the models again. To combat this Thompson‘s (2003) combined approach is seen as a useful framework against which to analyse the findings of this research. This approach is outlined in more detail in subsequent chapters.
The strategy appears similar to the transformative method that employs a theoretical lens to analyse data (Creswell, 2003), but does not in the case adopt mixed method approaches. This will instead involve considering decision making processes through a permutation of combined models. In this study I combine the models developed through the research of Allison (1999), Peterson (1976), Dahler-Larsen (1998) and Hardy (1990b, 1991, Hardy et al. 1983), which will inform the alternate templates. There is mainly use of qualitative methods, in line with naturalistic decision-making research, which focuses on the actual activity of decision makers rather than ―the decision event‖. Although such underlying values are often difficult for respondents to reconstruct (Beach
& Connolly, 2005) interviewing is considered to be a useful strategy to investigate such processes (Lipshitz, Klein, Orasanu, & Salas, 2001). It is therefore felt appropriate to interview ‗providers‘ involved in decision making about postgraduate programme evaluation, sampling purposively and theoretically. The intention of sampling purposively is to capture a semblance of heterogeneity in the population (Maxwell, 1996). Qualitative interview techniques are used in order to gain as rich a description as possible of the process and the actors‘ interpretation of it. In using qualitative interviewing as the main choice of methods, emphasis is placed upon discovering, recording and analysing participants attitudes to what guides the decision making process concerning evaluation models to be enacted within their organisation. The interviews are supported by analysis of secondary data, including documentary analysis from programme materials, national policy documents and other terms of reference, combined with literature review framing the problem within the fields of evaluation and naturalistic decision theory. The interview responses are then framed against the alternate templates (Langley, 1999) of organisational decision-making types for use within template analysis of the data (King, 1998, 2004). Template analysis is not a ―single, clearly delineated method‖ but rather a thematic organisation and analysis of textual data, based a list of codes (the templates) that represent themes in the data (King, 2004: 256). The initial template of theoretically determined nodes10 on which the analysis is based is outlined in the appendix. The final coding template is also included.
9 The authors analysed aspects of three different organisational contexts that shaped the decision processes, noting that the decision process itself, and not just the issue under discussion, affected a certain outcome (Maitlis & Ozcelik, 2004: 377).
10 A node refers to a place where an analytically determined category of data is assigned and stored in the place of a qualitative software programme (Richards, 2005).
15 1.7 Contribution
While much has been written on the subject of evaluation with regard to utilization, thought to be the resultant of good evaluation design, and additionally the factors that are believed to influence it, there has been much less focus on the underlying decision-making process and mechanisms that inform it.
With this in mind, the aim is to tie together research from the fields of programme evaluation, evaluation research and decision-making. The intention is also to develop existing frameworks and templates of decision-making, leading to greater understanding of the processes involved. This research is also an exploration of the impact of ideological positioning on the evaluation process. This is considered important with regard to how organisations can understand the processes that they develop and address any major issues arising from their values related to evaluation.