An important focus of this study is the attempt to explore why and how evaluations develop, while investigating the designed models. In order to understand these processes, this study draws on decision-making theory. While it is necessary to outline the major developments in decision-making research, it must be specified that the focus is on decision-making in action rather than on prescriptive decision modelling or experiments. Consequentially, it is considered vital to illuminate evaluation research with naturalistic decision-making theory (NDM) which attempts to discover the underlying attitudes and ideologies of those evaluating programmes, an area which has been suggested to have been under investigated (Holton III & Naquin, 2005). NDM research is considered useful as it focuses on how decisions are made, drawing on the perceptions of those involved in decision-making processes.
9
At the same time it is recognised that such organisational decision-making processes are complex, often unnoticed and not open to reductive descriptions.
Therefore a framework of decision-making models is applied, operating as alternate templates, which are grounded in decision making literature that can offer alternative perspectives to the process at hand. Such a framework is based on the application of multi-faceted models incorporating different strands of organisational decision making research, exemplified by Allison in his analysis of the handling of the Cuban Missile Crisis (1969, 1971; & Zelikow 1999).
However, such an approach has also been adapted and further applied within various educational settings (Ellstrom, 1983; Hardy, 1990b, 1991; Hardy, Langley, Mintzberg, & Rose, 1983; Peterson, 1976; Sergiovanni, 1979)6 based on the recognition that no one decision model will satisfactorily help analyse and explain all decision behaviour. As outlined earlier these models are supplemented by a template drawn from more recent developments in Institutional theory (Dahler-Larsen, 1998; W. R. Scott, 2003; Thompson, 1967, 2003), where the work of the former author has been applied generally to the field of evaluation.
Allison‘s three conceptual decision making models, are: ―rational actor, organizational behavior, and Government politics‖. The rational model paints the broader picture of a decision, including the search for an optimal choice. The organisational behaviour model focuses on the organisational rules and routines that produce information, options and action. Allison‘s third model investigates individual action and how perceptions and preferences are combined to influence decisions (1999: 392ff). These models were further adapted by Peterson (1976), where in particular, the final model is further divided into
―ideological bargaining, and pluralist bargaining‖. Allison recognised that a combination of these models or lenses should enable broader analysis of decision processes. As a result the third model is defined here as political bargaining. Research into decision processes since Allison‘s models were outlined allows for further development. A model based on developments in institutional theory allows for focus more on how environmental influence constitutively forms, develops and changes organisational identity. The models, then, rather than competing, combine to offer a more in depth understanding (Pfeffer, 1981b), helping to illuminate different ways that decisions are made.
As a result one could anticipate, like Dahler-Larsen (1998),that political and institutional models will overlap but that the impact over a longer period might be different.
These models are used to analyse the subunit decision-making concerning the adoption and implementation of a particular evaluation model for a programme.
6Valovirta (2002) can also be interpreted of conceiving these decision processes in a similar way, questioning whether evaluations are found to be more ‗academic‘, attempting to produce instrumental changes; bureaucratic, producing conceptual changes; or political, producing symbolic or legitimative utilization.
10
As has already been stated, focus on utilisation is delimited to how the intent to utilize information influences the type of evaluation model that is applied. This is considered to be part of the rationale or purpose for evaluation. This reiterates that the subsequent utilisation of the results of a particular evaluation is not under study in this project. This indicates that this concept it is important for its influence on and contribution to the purpose of the evaluation and how a particular model is chosen and implemented, but will not be fully investigated.
Hence the figure is not a causal model, but rather a framework to discover how the decision making process influences the resulting models chosen to evaluate school leadership training.
This model, outlined in figure 2, is developed to further direct focus to the organisational decision making process and related to the elements of evaluation decisions. Application of such a model in this study is at the micro level, where focus is placed upon the decision making concerning evaluation within HEI subunits responsible for postgraduate programmes for school leadership development. While the study is focused upon the micro level, it is also important to recognise that such decisions are taken within wider institutional and environmental contexts. With this in mind it is important to recognise how such models have been observed within a macro-perspective, for example Thompson (1967, 2003) and Scott‘s (2001, 2003) combined organisational models and their influence on decision-making. Figure 2 below therefore takes into the account the influence of actual and perceived environment demands on the decision makers. Understanding of the impact of these demands will be drawn from self-reports of interviewees as well as secondary data, including circulars, letters and planning documents etc. It is proposed that these demands can both influence evaluation design directly and indirectly and may depend upon the extent of the demands placed upon and perceived by decision makers.
This perception will be investigated through the study.
Figure 2: The analytical framework for understanding decisions about evaluation design
11
Figures 1 and 2 highlight the role of the decision makers in developing a design for an evaluation model that is considered to have an ultimate utilization purpose. It is reiterated that the final use of the evaluation findings is not under investigation per se; rather that investigation of the design process and model chosen is thought to give a clearer picture for analysis of how organisations respond to the demand to evaluate and plan to implement an evaluation. In the figure above the external and internal demands to evaluate are thought to influence the purpose of the evaluation (where external demands can notably influence internal demands as well). How these demands are interpreted within the subunit are part of the decision process to design a subsequent evaluation.
This is however not merely a political model, as is reflected in the use of alternate and inter-related templates to give a richer picture of the process. It is also recognised that the demand to evaluate might also be associated with a demand to evaluate in a particular way or with particular emphasis on indicators in mind. This too is thought to affect the process of decision-making and will be further investigated in the empirical research. This framework will be adapted and developed as the study progresses in line with the methodology outlined briefly in sections 1.7 and in more detail in chapter 5.
The basic idea behind the decision framework, then, is drawn from Allison (1999) and Peterson (1976), Dahler-Larsen (1998) and Thompson (2003) where information is drawn from decision and organisation theory and compared to current developments within the evaluation field. This also involves discussing forms of evaluation use, which are thought to influence decisions about evaluation purpose and design. The implications of such a coupling require investigation of the decision process in an organisation. The early design phases will be important as much as a discussion over who will be responsible and how they will carry it out. Therefore, although it is considered correct to ―distinguish between internal and external responsibility‖ for an evaluation at the arrangement, production and utilization phases (Vedung, 1997 in Dahler-Larsen, 2000), I also agree that this should be extended to look at the initiation of an evaluation and the broad influence across the phases towards use (Dahler-Larsen, 2000). This view appears even more necessary as evaluation is considered to take a much stronger role under NPM implemented throughout the public sector (Dahler-Larsen, 1998, 2005a, 2005b). This has seen evaluation developing from a typically one-off approach to a more institutionalised part of organisational routine (Hellstern, 1986 in Dahler-Larsen, 1998). This will require greater understanding of relationships and decision-making processes within organisations.
While organisational and decision-making theories are thought to illuminate the activity of evaluation there has been relatively little application of such research within the field of evaluation (Dahler-Larsen, 1998; Holton III & Naquin, 2005).
In recent years, however, there has been a growing interest in research towards these areas, particularly within the Danish research programme (Albaek, 1996;
Dahler-Larsen, 1998, 2004b, 2006b; Hansen, 2001, 2005a, 2005b; Krogstrup,
12
2006). While research elsewhere has continued to focus on the mechanics of improving evaluation and particularly utilisation,―[i]t is recognised that it is living social, political and organisational processes that form evaluations and decide whether evaluation results will be used‖7 (Dahler-Larsen, 1998: 25).
Many writers hypothesise the link from evaluation purpose and ultimate use, although one would particularly consider from rational approaches this link to be tightly coupled. But despite evaluation‘s relative rational basis, as a process it has not always matched up to general expectations. Therefore, evaluation is recognised to be both the ―child of rationalism and of rationalism‘s limitations‖
(Vedung, 1991 in, Dahler-Larsen, 1998: 23). Such reflections would appear to require deeper understanding of the framework within which evaluation develops. While this is recognised to be multifaceted and affected by many factors, certain areas will provide useful starting points for further study. With this in mind it is considered helpful to focus on one aspect of the evaluation process.