Investigating decisions about evaluation

In this current study when investigating the perceptions of decision behaviour with regard to evaluation and assessment within the subunits, it is considered necessary to explore how members consider the goals of evaluation at different levels. It will therefore be important to discuss the perception of degree of agreement within subunit, the degree of perceived agreement with wider organisational demands and the degree of perceived agreement with direct external demands. These intertwined relationships are presented in a model of goal agreement in figure 9 below:

Figure 9: Evaluative goal agreement across organisational levels

Degree of agreement within subunit

Perception of / Degree of agreement with wider evaluation system

Degree of agreement with external mandator

179

When applying these categories to the current context, the subunit under investigation can be understood to be the programme unit offering varying forms of postgraduate programme in school leadership development within the wider Higher Education Institution. The wider evaluation system will generally refer and apply to quality assurance systems that have been set up across the institution, but may also include other assessment strategies within specific institutes and/or faculties. It is recognised that these will be influenced by external pressures for evaluation at different levels. Influence from the subject field will also be considered important. As was seen in both chapters 2 and 3 one area of the field has tried to come to grips with is that of evaluation of output, more recently distinguished as impact. Despite striving towards greater understanding of cause and effect in this area (e.g. Guskey, 2000; Leithwood &

Levin, 2005) there does not appear yet a recognition that this has been ascertained. Many groups would challenge the validity of such search. It will be interesting to observe whether providers perceive a different reality that guides their action. Do they believe that cause and effect of their programmes is certain, and if not what guides their actions? As many programmes are offered to external mandators investigation will also be required as to how their goals for the programme and subsequent requirements for evaluation will influence the subunit and to what degree they are in agreement. Discussion should focus on how these processes develop. The interaction of these relationships is outlined in the figure below.

Figure 10: Evaluative goal agreement within context

For the purpose of this study, the categories of Stufflebeam and colleagues, outlined in section 5.1 have been adapted to take into account the application of broader decision models and more recent research into the design and implementation of evaluations outlined in this chapter. The new categories form the basis of research questions and areas for investigation with the various

Degree of agreement within subunit Organisational evaluation / QA system

Degree of agreement with external mandator

External evaluation pressures Requirements

for formal programmes

Matching to goals of subunit Demands

 local

 national

180

subunit members invited to take part in this study. The operationalisation and methodology associated with this process are outlined in the next chapter.

Table 8: Reapplication of Stufflebeam et al.’s categories of evaluation problems

151 Due to being focused upon the perceptions of programme providers rather than explaining the wider values of all stakeholders.

181

In summary, the different conceptual models build on a ―cluster of assumptions and categories‖ that influence analysis (Allison & Zelikow, 1999: 379). The models are not thought of as the main form of explanation of the decision making process at hand, but rather an analytical framework for understanding the participants‘ view of their organisational decision-making process. The models are seen as complimentary rather than mutually exclusive, that can offer competing conclusions as a result of their assumptions and propositions. Allison and Zelikow recognise that while Model 1 paints the broader picture of the decision that is made searching for an understanding of optimal choice and is a

―powerful, first approximation‖, Model 2 focuses on the organisational routines that produce the information, options and action. Model 3 is more detailed in understanding the individuals within the decision framework and how perceptions and preferences are combined (1999: 392ff). Together they should enable broader analysis of the evaluation process. Allison, however, opens for the possibility that alongside providing different answers, the models probably ask different questions. Recognizing the latter point would appear to partly allay concerns of whether it is possible to accept duplicate approaches concurrently (Pfeffer, 1981b). Model 4 recognises the complexity and ambiguity of organising; particularly the influence of the environment and the nature of how these demands might collectively be appraised within organisations that are governed by formal and informal pressures. The intention is to use these models to better understand the design processes that underlie the formation and implementation of evaluations. I now turn to the methodology for this study.

The data collected related to demands will be dealt with in chapter 7, while that concerning definitions and designs will be considered in chapter 8. Chapter 9 will include the data collected related to decision makers and decision making.

The next chapter deals with methodology for the study.

182