Evaluating Injury Prevention Initiatives - 2.4 EXAMPLES OF INJURY PREVENTION REVIEWS
2.4 EXAMPLES OF INJURY PREVENTION REVIEWS
An important review of childhood injury (Rivara, Beahler, Patterson,
Thompson, & Zavitkovsky, 1997) has been established as a link with the
Cochrane Collaboration, by the Harborview Injury Prevention Research Centre,
University of Washington (see http://weber.u.washington.edu/~hiprc/childinjury/).
In addressing the hierarchy of evidence issue, they listed studies
according to the following designs (ascending in order of strength of
- Ecologic study (the intervention or risk is exposed to an entire
population or group) although there is no control of the exposure some
non-intervention comparison groups may suffice for comparison, the timing and
the type of group provide the following bases for comparison:
- Ecological time (or time-tend) study: i.e., a before and after comparison of study.
- Ecological group study (comparing two or more groups concurrently)
- Ecological mixed study (between and within group comparisons over time)
- Case-control study, where the "cases" are people with the outcome of interest, versus "controls" who are people thought to be comparable to the cases, but without the outcome of interest (e.g. uninjured workers from the same workplace as injured "cases"). The two groups are analysed for differences in levels or types of exposure to possible risk factors or protective factors.
- Cohort study (comparisons of different interventions or exposures to risk factors, for samples of the same cohort, or panel of respondents)
- Controlled trials that were not randomized (convenience samples, or whole cohorts)
- Randomized controlled trials (including some community trials)
However, Rivara et al (1997) decided not to include the following study types of studies:
- Case series (anecdotal or uncontrolled cases)
- Laboratory studies (non-human or inanimate objects were the subjects)
It is a pity they discarded case studies in this way. Case studies can be useful in evaluations (see below section 2.4.4) provided that various conditions like Hill's criteria are met (see e.g., Kazi, 1996; Sechrest, et al., 1996; Rossi & Freeman, 1989).
While Rivara et al (1997) provide a model of how to categorise such studies,
the present review is more diverse in its search and is intended to be
primarily heuristic as well as to account for rigour. Thus the "mixed-criteria
approach" (Cooper, 1989) is most appropriate here. Certainly it is quite often
recommended in the evaluation literature (see Patton, 1990; Scriven, 1991), and
recently promoted in the health promotion literature (see Macdonald, 1996).
The chief concern for quantitative studies of causes and effects is that an
observed effect may be due to a factor or factors other than the one of primary
interest. Several study designs incorporate comparison groups to reduce the
chance of drawing false conclusions because of this type of problem. The study
design capable of providing the most rigorous defence against this is the
Randomised Control Trial (RCT), in which subjects are allocated at random to a
group to be exposed to the factor being studied (cases) or to a control group.
If possible, subjects and investigators are "kept in the dark" about whether
each subject is a "case" or a "control" until the end of the experiment (i.e. a
"double-blind" trial). RCT's have important strengths, and substantial
limitations. (see Campbell & Boruch, 1975) They tend to be expensive and
time-consuming (it is unusual to be able to use existing data). Many questions
cannot be studied by this method for ethical and logistic reasons. For Example,
The best evidence on the effectiveness of bicycle helmets comes from "weaker"
study designs than RCT's. A decision to conduct an RCT on this subject would
have to confront the ethics and practicality of establishing, by random
allocation, a group of "wearers", and a group of "non-wearers", and following
their injury experience for a long period (probably years).
Typical examples of this approach are the focus of the Cochrane Collaboration
(see e.g., Clunie, Ludbrook & Faris, 1995; Silagy & Jewell, 1994)
focusing on primary health care and general medical practice.
In injury prevention research and evaluation it is not always as applicable or
less systematically applied in various fields. For example, Zwerling et al
(1997, p. 164) found that in the occupational injury intervention studies they
reviewed, that: "randomized controlled trials are rare and also ... that the
quasi-experimental studies in the literature often use the weakest designs"
However, as Dannenberg, and Fowler (in press, 1998) point out the truly
double-blind, randomized control trial design is less frequently used in injury
prevention evaluation because of the ethical, financial and logistical
constraints. Indeed, these are common problems among evaluations of other
community based social interventions prevention (e.g., Covey, 1982; Hollister
& Hill, 1995).
Here the "case" person (or group) who has suffered an injury is compared with their
ilk who did not (the "controls") in terms of exposure to one or more suspected
"risk factors" for the injury. Armenian and Lilienfeld, (1994) provide an
excellent historical review of the use of case-control methods. This is one of
the oldest and most frequently used designs (see Solomon 1949). Because it only
relies on the existing selection of the case and the control, which are not
randomly allocated to groups, there are a number of plausible rival intervening
variables, including selection bias, which could explain the findings of such a
study. Despite its perceived limited scientific rigour, it is still a
useful evaluation design, because it enables some basic comparison to establish
change and relative effectiveness (see Armenian & Lilienfeld, 1994; Rossi
& Freeman, 1989).
The group of people enrolled in the study have various levels of exposure to a
suspected risk factor of interest (there may only be two levels: exposed or not
exposed). They are followed-up for some time, and the occurrence of injury is
recorded. Analysis is designed to look for association between exposure and
risk of injury.
A series of measurements or surveys of a group of similar individuals at risk
of injury, providing comparisons over time, before and after such risks might
occur, can be a useful technique for better understanding the impact of risk
factors, the needs of those at risk, and for pilot testing models or
interventions. It can be a preliminary basis for testing the efficacy of the
injury prevention (e.g. Berry, Gilmore & Geller, 1994) In some literature
this is called a panel survey (see GAO/PEMD 10.1.4 , 1991; Rossi ) Berry,
Gilmore and Geller (1994) provide two examples of time series analysis of
longitudinal case studies of automobile seat belt use.
Single case study (see Fisher, 1988; Hamel et al, 1993; Kazi, 1996; Sechrest, et al.,
1996; Rossi & Freeman, 1989; Smith & Everly, 1988; Yin, 1994) is often
associated with the Naturalistic inquiry and the qualitative approach to
evaluation (Lincoln & Guba, 1985; Guba & Lincoln, 1990; Patton, 1990).
Sechrest, Stewart, Stickle, & Sidani (1996) and Kratchowill (1979) argue
and demonstrate that the case study can have a high degree of validity and
interpretive power, given that they meet certain criteria of evidence.
Despite its complicated subjectivity and all the limitations of this method, it
has one important advantage over other evaluation designs, viz. its ability to
capture the imagination of the media, the public and decision makers by the
perception of the human interest, and the personal impact of the injury or the
injury prevention or control intervention. This is the stuff of the front-page
photo or anecdotal opening remarks for the executive summary or preface which
can grab the attention of the decision-makers and bring home the need for the
injury prevention intervention. Of course the findings could go for or against
the plans of the injury prevention or of the evaluation. That is the risk of
this kind of study, viz the inability to control the variables and the
interpretation of the results, as well as the ease with which it can be
dismissed by critics.
The ethical issues associated with this kind of study are probably more
significant than the usual evaluation, because of the exposure of the
participant to such attention (see Joint Committee on Standards, 1994; Sharp,
1994). Also there are difficulties in mixing case studies of at-risk
participants with the more management or efficiency oriented focus of some
evaluations (e.g. See Gray, Marshall, & Morris, 1997).
Sechrest et al. (1996, p 4 - 4) point out that:
"Remarkably little research has been done on the effectiveness
or persuasiveness of case studies. In fact, we know of only one such study, a
dissertation carried out by David Ametrano at the University of Michigan.
That is, we think, a highly unfortunate gap in our knowledge of a widely
used and potentially important methodology. Obviously more research needed if
we are to be able to write dependable specifications for producing persuasive
Based on their research, they argue that
there are at least three elements which can improve the persuasiveness of case
studies, as demonstrated in "classic" fundamental cases.
"First, it is true that cases that have come to be widely known
and accepted as generally plausible are interesting in their own right. ...
A second characteristic of classic case studies, most of which are in one
way or another meant to be probative, is that they either include or are
embedded in a context that represents a strong theoretical basis for inference.
A third characteristic of classic case studies is that they centred around
major, sometimes cataclysmic, events with very large and obvious
(Sechrest et al. , 1996, p 4-4)
Kazi (1996) also demonstrates that a combination of qualitative and quantitative data collection techniques can assist the Evaluator to make a more persuasive interpretation of the case study.