1.6 PROGRAM PLANNING & EVALUATION DESIGN LOGIC

• 1.6.1 Basic Steps in a System of Planning & Evaluation
• Fictitious Example

Most writers on evaluation emphasise the importance of preparation and planning for evaluation as part of program planning and design (e.g. Davis & Salasin, 1978; Rossi & Freeman, 1989; Winston, 1993; and Wholey, 1994). For most approaches to program management it is important to establish goals and objectives in the context of the injury prevention strategy and management policies. A technique often used in program evaluation to link the evaluation of a program with the total cycle of the program's planning, initiation, operation, outcomes, impact and decision processes is called "program logic" or implementation analysis. Program logic is used to analyse the overall framework of objectives into a logical causal relationship (see Lenne & Cleland, 1987). Generic outcome hierarchies can be developed which can be used as templates or guides to review the training program's implementation and check the results obtained in terms of the intended outcomes.

From the analysis of program logic it is possible to identify the timing and linkages in the sequence of events and cause and effect relations (see Kelly & McGrath, 1988). Indeed, the basic approach is akin to the "evidence diagrams" (e.g. Rivara, et al. 1997) often provided in association with regression analysis research.

Here the concept of program logic is introduced through a series of steps in an overview scheme of evaluation as a management and accountability system (see Figure 3). Then Section 2 will elaborate on the types of tools (see Charts I to VI) for implementing this scheme in terms of the linkages between the design of the injury prevention program, or control intervention, and the evaluation of it.

FIGURE 3: The Basic Steps of PROGRAM LOGIC ANALYSIS & EVALUATION

1.6.1 Basic Steps in a System of Planning & Evaluation

In essence, planning and evaluation form the processes of feedforward and feedback which are fundamental engines in any management system (see Checkland, 1981; Checkland & Scholes, 1990; Leeuw, Rist, & Sonnichsen, 1994; Mason, & Mitroff, 1981; Senge, 1990; Sharp, 1997a; Stacey, 1993a 1993b; 1993c; Stata, 1989).

A useful link between planning and evaluation comes from the program logic method of evaluation design, which involves the following steps:

• clarify the (1) purpose (including aims/goals) and (2) type of the program being reviewed whether it is focused on intervention or prevention or training per se, or more broadly educational, or advisory; versus an organisational culture or attitudinal change management program or on environmental change or regulatory change etc. (see the process in Figure 3 and a fictitious example of the product of Figure 3 in Figure 4);
• identify (3) who are the intended participants and describe characteristics relevant to subject (e.g., level of risk, demography, numbers and types of injuries prevalent);
• what do they (4) need? (e.g., analysis of risk, reduce exposure, increase knowledge, reduce incidence of injury) - needs analysis is a significant field in its own right which cannot be elaborated upon here (see McKillip, 1987; Percy-Smith, 1996; Siegel, Attkisson & Carson, 1978), but which will form an important basis for the conceptualisation of the evaluation and clarification of the methodology (see Charts II and III below)
• how can we demonstrate (5) that the injury prevention program or the environmental characteristics etc. influenced the participants and contributed to the reduction in need, or satisfaction of desired outcome?
• how can we determine whether any differences obtained were intended or unintended? This is an efficacy issue, pertaining to the investigation of cause and effect relationships, which require sophistication of the logic of the design of the program and the evaluation (see Chart I below).
• determine the (6) logic of the operational relationships between the components of the policy, objectives and operations of the program. Can we show the chain of inference or deduction between stages or components of the program or system in its implementation (see the processes in Figure 3 and a fictitious example in Figure 4 and follow the steps in Chart IV).
• develop and use a (7) relevant outcomes hierarchy to generate specific indicators or statements of outcomes for the program (e.g., specify attributes of the stakeholders and what their expectations are for what type of results, etc.);
  specify efficiency, effectiveness (efficacy) and appropriateness indicators in terms of program inputs, process, outputs and expected outcomes (see a fictitious example in Figure 5).

Above all there must be continuing interactive processes of monitoring and evaluation of performance of the participants and the program, e.g., use benchmarking (see Sharp, 1994) to compare the outcomes generated by the program with those from an outcomes hierarchy used in comparable organisations, or programs, to see whether there are major differences and to interpret why such differences might occur;

1. decide whether any of these differences are unsatisfactory and take appropriate steps to rectify;
2. identify what factors would have contributed to the successful achievement of these outcomes (by consultation with key informants and other stakeholders, and by reviewing the implementation processes conducted);
3. which of these factors were due to the operation of your program and can be demonstrated to be controllable?
4. which of these factors were not due to the operation of your program or did not appear to be controllable?
5. determine whether these factors can be brought under managerial control in the program;
6. what are the activities which can operate to produce success factors in future?
7. examine the program and outcomes again in terms of the generic hierarchy and determine the level of information required to monitor and manage the process in future.

Fictitious Example

The techniques of program logic and Goal Attainment Scaling are illustrated in Figures 4, 5 and 6, which deal with a fictitious example of how to clarify the objectives intended outcomes and performance indicators of a supposed program seeking to provide information advocating for the prevention of accidental injury in young people (especially males aged 15 to 20).
For purely a practical demonstration of the kinds of material which could be developed for injury prevention programs in the process of preparation for evaluation, the following goals are considered for an imaginary injury prevention program, say being considered for funding or evaluation by a state health department, targeted at the 15 to 20 year old males:

Goal 1: The injury prevention program will increase awareness of the extent and dimensions of youth injury (especially in males aged 15 to 20 yrs) over school holidays
Goal 2: The injury prevention program will be associated with a decreased incidence of severe injury in males aged 15 to 20 yrs by next budget

FIGURE 4
Analysis of the Suggested Needs & possible Risks underlying an Injury Prevention Program

Let us imagine the following Goals for a State Government Health Department:

Goal 1: Increased awareness of the extent and dimensions of youth injury (especially in males aged 15 to 20)
Goal 2: Reduced incidence of severe injury in males aged 15 to 20

FIGURE 5
Outcomes Hierarchy for Goal 1

7. Increased awareness of need for and Opportunity for targeted youth injury prevention programs
6. Young people have better access to and make use of higher quality targeted safety/injury prevention programs
5. Researchers and youth workers have better access to and make use of higher quality targeted safety/injury prevention programs
4. Injury Prevention projects & safety advocates improve the funding strategies, targeting and delivery of their advice, research and services.
3. State Health Agencies are better able to service and equip Policy Makers and Safety Advocates with necessary knowledge on how to target safety/injury prevention programs.
2. New models of injury risk and injury prevention and control are communicated by State Health Agency to Government, news & professional media
1. Advances are made in models (extent & dimensions) of youth injury by State Health Agency.