Systematic review to determine whether participation in a trial influences outcome.

(1)

doi:10.1136/bmj.330.7501.1175 2005;330;1175-

BMJ

Doris Tove Kristoffersen and Andrew David Oxman

Gunn Elisabeth Vist, Kåre Birger Hagen, P J Devereaux, Dianne Bryant, participation in a trial influences outcome Systematic review to determine whether

http://bmj.com/cgi/content/full/330/7501/1175 Updated information and services can be found at:

These include:

Data supplement

http://bmj.com/cgi/content/full/330/7501/1175/DC1

"Table"

References

http://bmj.com/cgi/content/full/330/7501/1175#otherarticles 5 online articles that cite this article can be accessed at:

http://bmj.com/cgi/content/full/330/7501/1175#BIBL

This article cites 53 articles, 11 of which can be accessed free at:

Rapid responses

http://bmj.com/cgi/eletter-submit/330/7501/1175 You can respond to this article at:

http://bmj.com/cgi/content/full/330/7501/1175#responses free at:

3 rapid responses have been posted to this article, which you can access for

service Email alerting

box at the top left of the article

Receive free email alerts when new articles cite this article - sign up in the

Topic collections

(367 articles) Systematic reviews (incl meta-analyses): examples

Articles on similar topics can be found in the following collections

Notes

To order reprints follow the "Request Permissions" link in the navigation box go to:

BMJ

To subscribe to

(2)

Papers

Systematic review to determine whether participation in a trial influences outcome

Gunn Elisabeth Vist, Kåre Birger Hagen, P J Devereaux, Dianne Bryant, Doris Tove Kristoffersen, Andrew David Oxman

Abstract

ObjectiveTo systematically compare the outcomes of participants in randomised controlled trials (RCTs) with those in comparable non-participants who received the same or similar treatment.

Data sourcesBibliographic databases, reference lists from eligible articles, medical journals, and study authors.

Review methodsRCTs and cohort studies that evaluated the clinical outcomes of participants in RCTs and comparable non-participants who received the same or similar treatment.

ResultsFive RCTs (six comparisons) and 50 cohort studies (85 comparisons) provided data on 31 140 patients treated in RCTs and 20 380 comparable patients treated outside RCTs. In the five RCTs, in which patients were given the option of participating or not, the comparisons provided limited information because of small sample sizes (a total of 412 patients) and the nature of the questions considered. 73 dichotomous outcomes were compared, of which 59 reported no statistically significant differences. For patients treated within RCTs, 10 comparisons reported significantly better outcomes and four reported significantly worse outcomes. Significantly heterogeneity was found (I²= 89%) among the comparisons of 73 dichotomous outcomes; none of our a priori explanatory factors helped explain this heterogeneity. The 18 comparisons of continuous outcomes showed no significant differences in heterogeneity (I²= 0%). The overall pooled estimate for continuous outcomes of the effect of participating in an RCT was not significant (standardised mean difference 0.01, 95%

confidence interval −0.10 to 0.12).

ConclusionNo strong evidence was found of a harmful or beneficial effect of participating in RCTs compared with receiving the same or similar treatment outside such trials.

Introduction

Properly conducted randomised controlled trials (RCTs) provide the strongest evidence of the effects of treatment.¹It is, however, controversial as to whether participants of such trials benefit directly or whether these studies are solely for the benefit of future patients. In addition, there is much scepticism about the applicability of the results to usual practice.²

Four reviews that considered whether it is beneficial or harmful to participate in RCTs drew varied conclusions.^3–6These reviews compared patients who were treated within trials with those treated outside the trials, regardless of differences between the clinical interventions or between the participants and non-participants. It is therefore uncertain whether the results

reflect the effects of participating in an RCT (trial effects), differences in the clinical interventions (treatment effects), or differences between participants and non-participants. We deter- mined whether the outcomes of participants in RCTs differed from those of comparable non-participants who received the same or similar treatment.

Methods

Our review was undertaken as a Cochrane methodology review (see Cochrane Library for fuller details of our methods and updated versions of the review).⁷

We included studies that compared participants in RCTs with comparable non-participants who received the same or similar treatment. We included observational studies and RCTs in which participation or the option of participation was randomly allocated.

Search strategies

We used seven strategies to identify relevant studies: consultation with experts; search of personal files; electronic searches of the Cochrane central register of controlled trials, Medline, Embase, the Cochrane methodology register, and PsycINFO; a review of reference lists from eligible articles; and a search of PubMed using the “related articles” feature and SciSearch. We also hand searched articles published in 2000 in five medical journals (BMJ,Annals of Internal Medicine,JAMA,Lancet, andNew England Journal of Medicine) to identify RCTs with over 200 patients and at least 100 eligible non-participants. Studies’ authors were con- tacted for data on the treatment and outcomes of eligible non-participants.

Assessment of study eligibility

Two reviewers independently assessed each article for eligibility.

Disagreements were resolved by discussion. A third reviewer was consulted when consensus could not be reached.

Data abstraction

Two reviewers independently abstracted data from eligible studies. Each study was assessed for selection bias (differences between participants and non-participants), detection bias, and exclusion bias (losses to follow-up). On the basis of the combined risks of the three biases, we grouped each comparison into overall quality groups (randomised, controlled comparisons, partially controlled comparisons, and poorly controlled comparisons) for analysis. Missing data were sought from the investigators.

A table giving the evidence profile of results is on bmj.com

(3)

Analysis

We compared the experimental group of the RCT with their respective eligible non-participants who received the treatment and the control group and eligible non-participants who received the control treatment outside of the trial. For each comparison, we analysed the main outcome, and, when reported, we analysed mortality separately. For all of the included comparisons we used the main outcome as reported by the investigators.

We analysed the dichotomous and continuous results separately.

The results are reported as relative risks with 95% confidence intervals, using adjusted estimates when available. For a summary of the dichotomous results in one table, we calculated the associated standard error from the natural logarithm of the unadjusted relative risk. Heterogeneity was assessed by ²test and the I²statistic using RevMan version 4.2.⁸For the unadjusted relative risk analysis, we used the Mantel-Haenszel test in RevMan. A fixed effect model was used to calculate summary statistics if no statistically significant (P < 0.10) heterogeneity was found among similar comparisons. For statistically heterogene- ous results we described the variation in the estimates and key explanatory factors; where possible relating the explanatory factors to observed differences in estimates of the effects of participation. We constructed a funnel plot to explore the possi- bility of publication bias.

Results

Overall, 55 studies, totalling 91 comparisons, met our inclusion criteria (fig 1). Forty one studies are still awaiting assessment, which currently cannot be included or excluded on the basis of the published information.

We identified five RCTs (six comparisons) in which patients were randomised according to whether they had the option to participate. These studies provided limited data because of their small sample sizes and the nature of the questions considered.

Two studies randomised 82 patients to “n of 1 trials” compared with standard practice—that is, randomised, double blind, multi- ple crossover comparisons of an active drug with a placebo in a single patient.^{9 10}One study (60 patients) measured spontane- ously self reported side effects in patients who had or had not been informed that they were in an RCT.¹¹ One study (227 patients) reported satisfaction among patients randomised to an RCT compared with patients randomised to a patient preference trial in which they had a choice of treatment.¹²Another study (43 patients) reported pain reduction among patients randomised to an RCT compared with those who were not invited to

participate.¹³None of these studies found significant differences in outcomes between patients treated in or outside RCTs.

Non-randomised cohort studies

We identified 50 cohort studies (85 comparisons) totalling 30 862 patients participating in RCTs compared with 20 246 patients treated outside RCTs.^14–63 Seventy comparisons comprised dichotomous outcomes, of which 12 reported adjusted estimates, and 15 comparisons comprised continuous outcomes.

We found significant heterogeneity (I²= 89.0%) among the results of comparisons with dichotomous main outcomes (fig 2);

these results were therefore not pooled. Of these 73 comparisons, 59 reported no significant differences between outcomes for patients treated in RCTs and those receiving similar treatments outside RCTs; 10 reported significantly better outcomes for patients treated in RCTs, and four reported significantly worse outcomes for patients treated in RCTs.

Identified citations (n=10 638)

Potentially relevant studies (n=339)

Assessed in detail for eligibility (n=153)

Included studies (n=55; 91 comparisons)

Clearly not relevant (n=10 299)

Excluded studies (n=186)

Excluded studies (n=98):

Different patients (n=8) Different treatments (n=22) Information not available (n=15) Awaiting further information (n=41) Other (n=12)

Fig 1 Selection process of eligible studies

Table 1 Summary of studies included in sensitivity analysis Number of

comparisons

Number of similar results

Number better in RCT

Number worse in RCT Selection bias:

Low risk 6 6 — —

No imbalance 22 19 1 2

Adjusted results 12 7 4 2

Partially controlled 18 15 2 1

Poorly controlled 33 30 3 —

Detection bias:

Low risk* 78 66 8 3

Partially met, similarly measured

5 3 2 1

High risk 8 8 — —

Exclusion bias:

No losses to follow up 38 32 4 2

1 person to 20%

lost to follow up

33 30 2 1

>20% loss to follow up 11 10 — 1

Unclear 9 4 4 1

Reasons for non-participation:

Refused 27 18 7 2

Refused because of preference

23 20 1 2

RCT versus preference trial

16 15 — 1

Not invited 8 8 — —

Treated by non-participating clinicians

1 1 — —

Different skills required for treatment:

Surgery and procedures

28 25 1 2

Drug treatment 22 17 4 1

Radiology 14 12 1 1

Usual care 9 8 1 —

Counselling and education

8 8 — —

Watchful waiting 7 6 — 1

Different clinical area:

Oncology 28 23 3 2

Obstetrics and gynaecology

14 12 2 1

Cardiology 13 10 2 1

Other internal medicine

11 9 2 —

Psychology and drug misuse

9 9 — —

Paediatrics 8 7 — 1

Respiration 2 2 — —

RCT=randomised controlled trial.

*Outcomes were measured in same way.

Papers

(4)

Study or subcategory Randomised Cooper 1997a¹² Cooper 1997b¹² Dahan 1986¹¹ Adjusted results Davis 1985²⁶ Feit 2000a²⁸ Feit 2000b²⁸ Mosekilde 2000a⁴⁴ Mosekilde 2000b⁴⁴ Schmoor 1996a⁵³ Schmoor 1996b⁵³ Schmoor 1996c⁵³ Schmoor 1996d⁵³ Schmoor 1996e⁵³ Schmoor 1996f⁵³ Williford 1993⁶⁰ Controlled comparisons Bhattacharya 1998¹⁸ Helsing 1998a³¹ Helsing 1998b³¹ Link 1991a³⁶ Link 1991b³⁶ Nagel 1998a⁴⁵ Nagel 1998b⁴⁵ Nicolaides 1994a⁴⁶ Nicolaides 1994b⁴⁶

Partially controlled comparisons Blichert-Toft 1988a²⁰ Blichert-Toft 1988b²⁰ CASS 1984a²¹ CASS 1984b²¹ Chauhan 1991²² Clagett 1984a²⁴ Clagett 1984b²⁴ Creutzig 1993a²⁵ Creutzig 1993b²⁵ Forbes 2000²⁹ Henshaw 1993a³² Henshaw 1993b³² Liu 1998a³⁷ Liu 1998b³⁷

Martinez-Amenos 1990a³⁹ Martinez-Amenos 1990b³⁹ Rigg 2000a⁵⁰ Rigg 2000b⁵⁰ Strandberg 1995⁵⁴ Poorly controlled comparisons Balmukhanov 1989a¹⁴ Balmukhanov 1989b¹⁴ Baum 1979¹⁵ Berglund 1997¹⁷ Bijker 2000a¹⁹ Bijker 2000b¹⁹ Chilvers 2001a²³ Chilvers 2001b²³ Edsmyr 1978²⁷ Forssell 1989³⁰ Kieler 1998³³ King 1997a³⁴ King 1997b³⁴ Lidbrink 1995³⁵ MACESG 1992a³⁸ MACESG 1992b³⁸ Moertel 1984⁴³ Playforth 1988⁴⁹ Rosen 1987a⁵¹ Rosen 1997b⁵¹ Sullivan 1982a⁵⁵ Sullivan 1982b⁵⁵ Sullivan 1982c⁵⁵ Urban 1999⁵⁶ Wetzner 1979⁵⁸ Wikdahl 1992⁵⁹ Yamamoto 1992a⁶¹ Yamamoto 1992b⁶¹ Yersin 1996⁶² Young 1996⁶³

Log (relative risk) (SE)

Relative risk (random) (95% CI) 0.24 (0.21)

-0.14 (0.39) 2.20 (1.47)

-0.94 (0.15) 0.16 (0.01) -0.06 (0.02) 0.14 (0.11) 0.28 (0.42) 0.02 (0.04) 0.01 (0.03) -0.33 (0.07) -0.06 (0.10) 0.18 (0.05) -0.00 (0.09) -0.51 (0.03)

-0.12 (0.49) -0.07 (0.14) -0.08 (0.06) -0.11 (0.37) 0.06 (0.13) 0.24 (0.80) 0.75 (1.53) 0.15 (0.32) -1.14 (0.64)

0.43 (0.45) 1.03 (0.51) 0.06 (0.23) 0.07 (0.20) 0.02 (0.39) -0.29 (0.56) -1.10 (1.59) 1.47 (1.03) 0.12 (0.36) -0.58 (0.48) 1.68 (0.60) -0.53 (0.90) -1.09 (1.21) -0.13 (0.90) -0.31 (0.14) -0.03 (0.17) -1.46 (0.61) 0.28 (0.61) -0.52 (0.14)

-0.91 (0.75) -0.10 (0.18) -1.30 (0.66) 0.49 (0.28) -0.42 (0.54) 0.78 (0.45) -0.07 (0.11) 0.12 (0.11) -0.80 (1.14) -0.05 (0.63) -0.05 (0.63) 0.25 (0.39) -0.16 (0.36) -0.34 (0.26) -0.52 (0.60) 0.74 (0.73) 0.31 (0.21) 0.49 (0.58) -0.21 (0.07) -0.17 (0.06) 1.91 (1.39) -2.60 (1.34) 0.38 (1.05) 0.32 (0.24) 0.86 (0.50) -0.03 (0.71) -0.25 (0.18) -0.16 (0.14) 0.00 (0.45) -0.06 (0.20)

Weight (%) 67.57 29.78 2.65

6.64 9.90 9.87 7.85 1.97 9.56 9.71 8.83 8.09 9.43 8.47 9.67

4.90 21.50 29.08 7.50 22.14

2.03 0.58 9.23 3.04

3.89 3.23 9.75 11.58

4.84 2.72 0.38 0.88 5.50 3.59 2.42 1.13 0.64 1.13 15.08 13.29 2.30 2.33 15.32

0.77 6.02 0.97 3.77 1.39 1.90 8.26 8.39 0.35 1.06 1.06 2.38 2.65 4.19 1.16 0.80 5.24 1.22 9.47 9.74 0.24 0.25 0.41 4.66 1.58 0.85 6.22 7.44 1.91 5.64

Relative risk (random) (95% CI) 1.27 (0.85 to 1.90) 0.87 (0.40 to 1.86) 9.00 (0.51 to 160.07)

0.39 (0.29 to 0.52) 1.17 (1.15 to 1.19) 0.94 (0.91 to 0.97) 1.15 (0.93 to 1.42) 1.32 (0.58 to 3.02) 1.03 (0.95 to 1.11) 1.01 (0.95 to 1.07) 0.72 (0.62 to 0.83) 0.94 (0.77 to 1.14) 1.20 (1.09 to 1.31) 1.00 (0.84 to 1.18) 0.60 (0.56 to 0.64)

0.89 (0.34 to 2.30) 0.93 (0.70 to 1.22) 0.92 (0.83 to 1.03) 0.89 (0.43 to 1.86) 1.07 (0.82 to 1.39) 1.27 (0.27 to 6.06) 2.11 (0.10 to 42.37) 1.16 (0.61 to 2.19) 0.32 (0.09 to 1.12)

1.53 (0.63 to 3.73) 2.79 (1.04 to 7.53) 1.06 (0.67 to 1.68) 1.07 (0.73 to 1.59) 1.02 (0.47 to 2.21) 0.75 (0.25 to 2.23) 0.33 (0.01 to 7.52) 4.33 (0.58 to 32.48) 1.13 (0.55 to 2.29) 0.56 (0.22 to 1.43) 5.36 (1.66 to 17.28) 0.59 (0.10 to 3.44) 0.34 (0.03 to 3.64) 0.87 (0.15 to 5.11) 0.73 (0.56 to 0.97) 0.97 (0.70 to 1.35) 0.23 (0.07 to 0.77) 1.32 (0.40 to 4.37) 0.59 (0.45 to 0.78)

0.40 (0.09 to 1.76) 0.91 (0.63 to 1.30) 0.27 (0.07 to 0.99) 1.63 (0.94 to 2.83) 0.65 (0.23 to 1.88) 2.18 (0.91 to 5.25) 0.93 (0.75 to 1.16) 1.13 (0.91 to 1.39) 0.45 (0.05 to 4.16) 0.95 (0.28 to 3.29) 0.95 (0.28 to 3.29) 1.28 (0.60 to 2.75) 0.85 (0.42 to 1.73) 0.71 (0.43 to 1.18) 0.59 (0.18 to 1.91) 2.10 (0.50 to 8.81) 1.37 (0.90 to 2.07) 1.63 (0.52 to 5.10) 0.81 (0.70 to 0.93) 0.84 (0.75 to 0.95) 6.75 (0.45 to 102.39)

0.2 0.5

Favours being in randomised controlled trial Favours being outside randomised controlled trial

0 2 5

Fig 2 Results of dichotomous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

(5)

Figure 3 shows the results of the 18 comparisons with continuous main outcomes. We found no significant heterogeneity (I²= 0%). The pooled estimate found no differences in outcomes for patients treated in and outside RCTs (standardised mean difference 0.01, 95% confidence interval −0.10 to 0.12).

In 17 studies (32 comparisons) with data on mortality (fig 4), we found significant heterogeneity (I²= 88.8%); the results were therefore not pooled. In 24 of the 32 comparisons we found no significant difference in mortality. Four comparisons reported a significant lower risk of mortality for patients treated in RCTs and four comparisons reported a significantly higher risk of mortality.

Separate subgroup analyses could not explain the observed heterogeneity by the different types of eligible non-participants, treatments, clinical specialities, or study quality (selection bias, detection bias, and exclusion bias). The table summarises the studies included in the sensitivity analysis.

The funnel plot of the dichotomous comparisons showed no asymmetry (fig 5), indicating a low risk of publication bias.

Discussion

Our systematic review found no strong evidence of a harmful or beneficial trial effect of participating in randomised controlled trials (RCTs). The five included RCTs provided limited evidence

because of their small sample sizes and the nature of the questions they considered, but they did show that it is possible to consider questions about the effects of participating in RCTs by using randomised designs. Our interpretation of the 50 non-randomised cohort studies was limited by the quality and size of the comparisons and the wide variations in participants, clinical interventions, and outcomes. Most of the 85 non-randomised cohort comparisons found no statistically significant differences, although 10 reported better outcomes for patients in RCTs and four reported better outcomes for patients outside of RCTs.

Previous reviews that considered a less precise question than the one we evaluated drew varied conclusions. For example, one identified 14 articles reporting data from 21 trials and concluded that, if anything, randomised trials tend to have beneficial rather than harmful effects.⁴Another review included seven of these 14 articles and 17 additional articles.⁶ Only eight of the studies compared trial patients with non-trial patients who met the same eligibility criteria, and it was only possible to separate treatment effects from trial effects in three of these trials. A further review found 10 comparisons on survival or quality of life of patients treated in RCTs for life threatening illnesses (eight were cancer treatments) with those treated outside RCTs.⁵It found evidence of longer survival in participants, but the authors were not confi- dent of the results.

Study or subcategory Randomised Bergmann 1994¹³ Mahon 1996⁹ Mahon 1999¹⁰ Subtotal (95% CI)

Test for heterogeneity: χ²=3.36, df=2, P=0.19, I ²=40.5%

Test for overall effect: z=0.48, P=0.63 Controlled comparison

McKay 1995a⁴¹ McKay 1995b⁴¹ McKay 1998a⁴² McKay 1998b⁴² Rovers 2001a⁵² Rovers 2001b⁵² Subtotal (95% CI)

Test for heterogeneity: χ²=3.98, df=5, P=0.55, I ²=0%

Test for overall effect: z=0.34, P=0.74 Partially controlled comparison McCaughey 1998⁴⁰

Villamaria 1997a⁵⁷ Villamaria 1997b⁵⁷ Subtotal (95% CI)

Test for heterogeneity: χ²=5.61, df=2, P=0.06, I ²=64.3%

Test for overall effect: z=0.55, P=0.58 Poorly controlled comparison Bedi 2000a¹⁶

Bedi 2000b¹⁶ Paradise 1984a⁴⁷ Paradise 1984b⁴⁷ Paradise 1990a⁴⁸ Paradise 1990b⁴⁸ Subtotal (95% CI)

Test for overall effect: z=0.01, P=0.99 Total (95% CI)

Test for overall effect: z=0.21, P=0.84 No

Standardised mean difference (fixed)

(95% CI)

18 12 31 61

16 18 49 41 93 94 311

6 30 30 66

52 51 38 35 52 47 275

713

No

25 9 30 64

52 16 29 15 36 97 245

20 8 16 44

140 80 44 34 46 67 411

764 In randomised controlled trials

Mean (SD)

-22.10 (31.00) -12.00 (29.00) 7.00 (65.00)

3.38 (5.70) 6.89 (9.05) 1.61 (4.29) 2.61 (4.43) 36.00 (19.68) 70.00 (24.73)

10.60 (4.30) -37.57 (0.71) -37.29 (0.59)

15.20 (11.60) 14.80 (10.10) 0.76 (1.01) 2.66 (2.30) 10.18 (8.43) 13.78 (9.17)

Outside of randomised controlled trials

Mean (SD)

-5.30 (34.00) -3.00 (53.00) -8.00 (63.00)

4.90 (7.90) 7.88 (9.97) 2.00 (4.05) 1.60 (3.38) 30.00 (15.31) 71.00 (20.10)

5.10 (5.50) -37.22 (0.65) -37.38 (0.84)

14.40 (9.80) 14.00 (9.30) 1.05 (1.22) 2.41 (2.27) 11.19 (7.23) 13.41 (9.78)

Weight (%)

3.04 1.54 4.55 9.13

3.66 2.54 5.47 3.28 7.72 14.34 37.02

1.25 1.86 3.13 6.24

11.39 9.35 6.08 5.18 7.32 8.30 47.62

100.00

Standardised mean difference (fixed)

(95% CI)

-0.50 (-1.12 to 0.11) -0.21 (-1.08 to 0.66) 0.23 (-0.27 to 0.74) -0.09 (-0.44 to 0.27)

-0.20 (-0.76 to 0.36) -0.10 (-0.78 to 0.57) -0.09 (-0.55 to 0.37) 0.24 (-0.36 to 0.83) 0.32 (-0.07 to 0.71) -0.04 (-0.33 to 0.24) 0.03 (-0.15 to 0.21)

1.01 (0.05 to 1.97) -0.49 (-1.28 to 0.30)

0.13 (-0.48 to 0.74) 0.12 (-0.31 to 0.55)

0.08 (-0.24 to 0.40) 0.08 (-0.27 to 0.43) -0.25 (-0.69 to 0.18) 0.11 (-0.36 to 0.58) -0.13 (-0.52 to 0.27) 0.04 (-0.33 to 0.41) 0.00 (-0.15 to 0.16)

0.01 (-0.10 to 0.12)

-4 -2

0 2 4

Fig 3 Results of continuous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

Papers

(6)

Our review differs from these reviews in several ways, includ- ing the scope and comprehensiveness of our search, our method of analysis, and the question we asked, which controlled for differences in the effects of different interventions and differences between participants and non-participants. (See bmj.com for an evidence profile of our results according to the Grading of Rec- ommendations Assessment, Development, and Evaluation (GRADE) Working Group.) Our results are based mainly on comparisons of cohorts and are subject to the usual uncertainty associated with observational studies.¹Additionally, we could not explain the significant heterogeneity between studies, which affects our confidence in the results and reduces the overall quality of information. Other relevant studies apart from those included in this review may exist, as indicated by the number of studies awaiting assessment and the difficulty we and others had

in searching for this type of study in electronic databases. As we did not find evidence of publication bias, it is unlikely that the studies that we failed to identify would provide strong evidence of either harmful or beneficial effects.

An important corollary of this finding is that it counters sug- gestions that the results of RCTs cannot be applied to usual clinical practice, because most of the studies found no significant difference in outcomes for participants of RCTs compared with comparable non-participants who received similar treatment.

In most cases, RCTs seem to provide estimates of treatment effects that are applicable to comparable patients who receive similar interventions in usual clinical practice. In addition to being informed about the risks and harms of an intervention when invited to participate in RCTs, patients can be told that, independently of the effects of the interventions being compared, participating in a trial is likely to result in similar outcomes to patients who receive the same or similar treatment outside of the trial.

We thank Dave Sackett for initiating this review, for his contributions to the protocol, and for his support; Lena Nordheim for help with the search strategy; and the following for additional information about their studies: S M Albert, B J Barrett, J M Bennett, S B Black, S S Brehm, G Berglund, J F Bergmann, N Bijker, M Blichert-Toft, L Boros, J G Cairncross, K M Carroll, D W Chadwick, S P Chauhan, C Chilvers, G P Clagett, C J Cohen, K Collins, J S Cooper, K G Cooper, V Cottin, U Creutzig, S Davis, E P DiMagno, J van Eys, P J Ferrone, G M Forbes, J Frisell, B Glimelius, C Gluud, F Guillemin, J C Hauth, M Helsing, D Hellberg, J M Henderson, A M Hutter, L P Jensen, J J C Jonker, T Karrison, S B King, D Kjar, R A Kronmal, R W Licht, E Lid- brink, W F Liu, J K Madsen, J L Mahon, M A Malangoni, C Marcocci, J R McKay, M P Mourits, C Moynihan, G R Mundy, C D Naylor, M G Neuwirth, J R O’Fallon, J L Paradise, A Pinchera, M J Playforth, A V Pollock, P D Rokke, M A Rosen, M M Rovers, R G Rychtarik, B Schmidt, C Schmoor, B E Sha, J H Shelhamer, B G Stegmayr, C A Stiller, J M Stone, R Takolander, Study or subcategory

Adjusted mortality Davis 1985²⁶ Feit 2000a²⁸ Feit 2000b²⁸ Schmoor 1996a⁵³ Schmoor 1996b⁵³ Schmoor 1996c⁵³ Schmoor 1996d⁵³ Schmoor 1996e⁵³ Schmoor 1996f⁵³ Unadjusted mortality CASS 1984a²¹ CASS 1984b²¹ Helsing 1998a³¹ Helsing 1998b³¹ Kieler 1998³³ King 1997a³⁴ King 1997b³⁴ Lidbrink 1995³⁵ Link 1991a³⁶ Link 1991b³⁶ Moertel 1984⁴³ Nagel 1998a⁴⁵ Nagel 1998b⁴⁵ Nicolaides 1994a⁴⁶ Nicolaides 1994b⁴⁶ Playforth 1988⁴⁹ Rigg 2000a⁵⁰ Rigg 2000b⁵⁰ Strandberg 1995⁵⁴ Sullivan 1982a⁵⁵ Sullivan 1982b⁵⁵ Sullivan 1982c⁵⁵ Urban 1999⁵⁶

Log (relative risk) (SE)

Relative risk (random) (95% CI)

-0.94 (0.15) -0.06 (0.02) 0.16 (0.01) 0.00 (0.05) 0.17 (0.04) -0.13 (0.09) 0.43 (0.09) 0.26 (0.05) 0.10 (0.09)

0.06 (0.23) 0.07 (0.20) -0.07 (0.14) -0.08 (0.06) -0.05 (0.63) 0.25 (0.39) -0.16 (0.37) -0.34 (0.26) 1.19 (1.39) 0.00 (1.97) 0.31 (0.21) 0.24 (0.80) 0.75 (1.53) 0.15 (0.32) -1.14 (0.64) 0.49 (0.58) -1.46 (0.61) 0.28 (0.61) -0.52 (0.14) 0.15 (1.50) -0.90 (1.48) -0.10 (1.60) 0.32 (0.24)

Weight (%)

7.10 13.09 13.16 11.95 12.57 10.13 9.86 11.99 10.13

6.87 8.53 12.20 19.32 1.31 3.13 3.52 5.98 0.28 0.14 7.88 0.84 0.23 4.24 1.27 1.52 1.38 1.39 12.45

0.24 0.25 0.21 6.81

Relative risk (random) (95% CI)

1.06 (0.67 to 1.68) 1.07 (0.73 to 1.59) 0.93 (0.70 to 1.22) 0.92 (0.83 to 1.03) 0.95 (0.28 to 3.29) 1.28 (0.60 to 2.75) 0.85 (0.42 to 1.73) 0.71 (0.43 to 1.18) 3.28 (0.22 to 49.81) 1.00 (0.02 to 47.78) 1.37 (0.90 to 2.07) 1.27 (0.27 to 6.06) 2.11 (0.10 to 42.37) 1.16 (0.61 to 2.19) 0.32 (0.09 to 1.12) 1.63 (0.52 to 5.10) 0.23 (0.07 to 0.77) 1.32 (0.40 to 4.37) 0.59 (0.45 to 0.78) 1.16 (0.06 to 22.10) 0.41 (0.02 to 7.47) 0.90 (0.04 to 20.82) 1.38 (0.87 to 2.19)

0.2 0.5

0 2 5

Fig 4 Comparison of mortality between participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

Relative risk (fixed)

Standard error (log relative risk)

0.1 0.2 0.5 1 2 5 10

2.0 1.6 1.2 0.8 0.4 0

Fig 5 Funnel plot of dichotomous comparisons in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

(7)

I F Tannock, P Vestergaard, B Ward, C Weijer, D J Weisdorf, D G Wyse, B Yersin, and V L Yu.

Contributors: GEV carried out the literature search. GEV and KBH selected the studies. GEV, KBH, PJD, and DB extracted the data. GEV, DTK, and ADO analysed the data. GEV, KBH, PJD, DB, and ADO drafted the manu- script. GEV, KBH, PJD, and ADO developed the protocol and review. DB hand searched the five medical journals. GEV is guarantor for the paper.

Funding: Norwegian Health Services Research Centre, McMaster University, and the Nuffield Trust.

Competing interests: None declared.

Ethical approval: Not required.

1 Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in health- care trials (Cochrane methodology review). Issue 4. Oxford: Update Software, 2002.

2 Rothwell PM. External validity of randomised controlled trials: “To whom do the results of this trial apply?”Lancet2005;365:82-93.

3 Stiller CA. Centralised treatment, entry to trials and survival.Br J Cancer1994;70:352- 62.

4 Braunholtz DA, Edwards SJL, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect.”J Clin Epidemiol2001;54:217-24.

5 The Emergency Care Research Institute 2002. Patients’ reasons for participation in clinical trials and effect of trial participation on patient outcomes. www.ecri.org/

Patient_Information/Patient_Reference_Guide/evidence.pdf (accessed Nov 2004).

6 Peppercorn JM, Weeks JC, Cook EFC, Joffe S. Comparison of outcomes in cancer patients treated within and outside clinical trials: conceptual framework and structured review.Lancet2004;363:263-70.

7 Vist GE, Hagen KB, Devereaux P, Bryant D, Kristoffersen DT, Oxman AD. Outcomes of patients who participate in randomised controlled trials compared to similar patients receiving similar interventions who do not participate (Cochrane methodology reviews). Issue 1. Oxford: Update Software, 2005.

8 Higgins JP, Thompson SG. Quantifying heterogeneity in meta-analysis.Stat Med 2002;21:1539-58.

9 Mahon J, Laupacis A, Donner A, Wood T. Randomised study of n of 1 trials versus standard practice.BMJ1996;312:1069-74.

10 Mahon JL, Laupacis A, Hodder RV, McKim DA, Paterson NAM, Wood TE, et al. Theo- phylline for irreversible chronic airflow limitation. A randomized study comparing n of 1 trials to standard practice.Chest1999;115:38-48.

11 Dahan R, Caulin C, Figea L, Kanis JA, Caulin F, Segrestaa JM. Does informed consent influence therapeutic outcome? A clinical trial of the hypnotic activity of placebo in patients admitted to hospital.BMJ1986;293:363-4.

12 Cooper KG, Grant AM, Garratt AM. The impact of using a partially randomised patient preference design when evaluating alternative managements for heavy menstrual bleeding.Br J Obstet Gynaecol1997;104:1367-73.

13 Bergmann JF, Chassany O, Gandiol J, Deblois P, Kanis JA, Segrestaa JM, et al. A randomised clinical trial of the effect of informed consent on the analgesic activity of placebo and naproxen in cancer pain.Clin Trials Metaanal1994;29:41-7.

14 Balmukhanov SB, Beisebaev AA, Aitkoolova ZI, Mustaphin JS, Philippenko VI, Rismu- hamedova RS, et al. Intramural and parametrial infusion of metronidazole in the radiotherapy of uterine cervix cancer: preliminary report.Int J Radiation Oncol Biol Phys 1989;16:1061-3.

15 Baum E, Sather H, Nachman J, Seinfeld J, Kritvit W, Leikin S, et al. Relapse rates following cessation of chemotherapy during complete remission of acute lymphocytic leukemia: a report from childrens cancer study group.Med Pediatr Oncol1979;7:25-34.

16 Bedi N, Chilvers C, Churchill R, Dewey M, Duggan C, Fielding K, et al. Assessing effec- tiveness of treatment of depression in primary care: partially randomised preference trial.Br J Psychol2000;177:312-8.

17 Berglund G, Bolund C, Gustafsson UL, Sjoden PO. Is the wish to participate in a cancer rehabilitation program an indicator of the need? Comparisons of participants and non-participants in a randomized study.Psychooncology1997;6:35-46.

18 Bhattacharya S, Cameron IM, Mollison J, Parkin DE, Abramovich DR, Kitchener HC.

Admission-discharge policies for hysteroscopic surgery: a randomised comparison of day case with in-patient admission.Eur J Obstet Gynecol Reprod Biol1998;76:81-4.

19 Bijker N, Peterse JL, Fentiman TS, Julien JP, Hart AAM, Avril A, et al. Effects of patient selection on the applicability of results from a randomised clinical trial (EORTC

10853) investigating breast-conserving treatment for DCIS. Breast conserving therapy for ductal carcinoma in situ. PhD Thesis, 2000.

20 Blichert-Toft M, Brincker H, Andersen JA, Andresen KW, Axelsson CK, Mouridsen HT, et al.. A Danish randomized trial comparing breast-conserving therapy with mastectomy in mammary carcinoma: preliminary results. Acta Oncologica 1988;27:671-7.

21 Coronary artery surgery study (CASS): a randomised trial of coronary artery bypass surgery. Comparability of entry characteristics and survival in randomized patients and nonrandomized patients meeting randomization criteria.J Am Coll Cardiol1984;3:114- 28.

22 Chauhan SP, Rutherford SE, Hess LW, Morrison JC. Prophylactic intrapartum amnio- infusion for patients with oligohydramnios: a prospective randomized study.J Reprod Med1991;37:817-20.

23 Chilvers C, Dewey M, Fielding K, Gretton V, Miller P, Palmer B, et al. Antidepressant drugs and generic counselling for treatment of major depression in primary care: randomised trial with patient preference arms.BMJ2001;322:772-5.

24 Clagett GP, Youkey JR, Bringham RA, Orecchia PM, Salander JM, Collins GJ, et al.

Asymptomatic cervical bruit and abnormal ocular pneumoplethysmography: a prospective study comparing two approaches to management.Surgery1984;96:823-30.

25 Creutzig U, Ritter J, Zimmermann M, Schellong G. Does cranial irradiation reduce the risk of bone marrow relapse in acute myelogenous leukemia? Unexpected results of the childhood acute myelogenous leukemia study BFM-87.J Clin Oncol1993;11:279-86.

26 Davis S, Wright PW, Schulman SF, Hill LD, Pinkham RD, Johnson LP, et al. Participants in prospective, randomized clinical trials for resected non-small cell lung cancer have improved survival compared with nonparticipants in such trials. Cancer 1985;56:1710-8.

27 Edsmyr F, Esposti PL, Johansson B, Strindberg B. Clinical experimental randomized study of 2.6-CIS-diphenylhexamethylcyclotetrasiloxane and estramurine-17- phosphate in the treatment of prostatic carcinoma.J Urol1978;120:705-7.

28 Feit F, Brooks MM, Sopko G, Keller NM, Rosen A, Krone R, et al. Long-term clinical outcome in the bypass angioplasty revascularization investigation registry: comparison with the randomized trial.Circulation2000;101:2795-802.

29 Forbes GM, Collins BJ. Nitrous oxide for colonoscopy: a randomized controlled study.

Gastrointest Endosc2000;51:271-7.

30 Forssell C, Takolander R, Bergqvist D, Johansson A, Persson NH. Local versus general anaesthesia in carotid surgery. A prospective, randomised study.Eur J Vasc Surg 1989;3:503-9.

31 Helsing M, Bergman B, Thaning L, Hero U, for the Joint Lung Cancer Study Group.

Quality of life and survival in patients with advanced non-small cell lung cancer receiving supportive care plus chemotherapy with carboplatin and etoposide or supportive care only. A multicentre randomised phase III trial.Eur J Cancer1998;34:1036-44.

32 Henshaw RC, Naji SA, Templeton AA. Comparison of medical abortion with surgical vacuum aspiration: women’s preferences and acceptability of treatment. BMJ 1993;307:714-7.

33 Kieler H, Hellberg D, Nilsson S, Waldenstrøm U, Axelsson O. Pregnancy outcome among non-participants in a trial on ultrasound screening.Ultrasound Obstet Gynecol 1998;11:104-9.

34 King III SB, Barnhart HX, Kosinski AS, Weintraub WS, Lembo NJ, Petersen JY, et al.

Angioplasty or surgery for multivessel coronary artery disease: comparison of eligible registry and randomized patients in the EAST trial and influence of treatment selection on outcomes.Am J Cardiol1997;79:1453-9.

35 Lidbrink E, Frisell J, Rosendahl I, Rutqvist LE. Nonattendance in the Stockholm mam- mography screening trial: relative mortality and reasons for nonattendance.Breast Cancer Res Treat1995;35:267-75.

36 Link MP, Goorin AM, Horowitz M, Meyer WH, Belasco J, Baker A, et al. Adjuvant chemotherapy of high-grade osteosarcoma of the extremity. Updated results of the multi-institutional osteosarcoma study.Clin Orthop Relat Res1991;270:8-14.

37 Liu WF, Harrington T. The need for delivery room intubation of thin meconium in the low risk newborn: a clinical trial.Am J Perinatol1998;15:675-82.

38 Mayo Asymptomatic Carotid Endarterectomy Study Group. Results of a randomized controlled trial of carotid endarterectomy for asymptomatic carotid stenosis.Mayo Clinic Proc1992;67:513-8.

39 Martinez-Amenos A, Ferre MLF, Vidal CM, Rocasalbas JA. Evaluation of two educative models in primary care hypertension programme.J Hum Hypertens1990;4:362-4.

40 McCaughey ES, Mulligan J, Voss LD, Betts PR. Randomised trial of growth hormone in short normal girls.Lancet1998;351:940-4.

41 McKay JR, Alterman AI, McLellan AT, Snider EC, O’Brian CP. Effect of random versus nonrandom assignment in a comparison of inpatient and day hospital rehabilitation for male alcoholics.J Consult Clin Psychol1995;63:70-8.

42 McKay JR, Alterman AI, McLellan AT, Boardman CR, Mulvaney FD, O’Brian CP. Ran- dom versus nonrandom assignment in the evaluation of treatment for cocaine abusers.

J Consult Clin Psychol1998;66:697-701.

43 Moertel CG, Childs DS, O’Fallon JR, Holbrook MA, Schutt AJ, Reitemeier RJ.

Combined 5-fluorouracil and radiation therapy as a surgical adjuvant for poor prognosis gastric carcinoma.J Clin Oncol1984;2:1249-54.

44 Mosekilde L, Beck-Nielsen H, Sørensen OH, Nielsen SP, Charles P, Vestergaard P, et al.

Hormonal replacement therapy reduces forearm fracture incidence in recent postmenopausal women—results of the Danish osteoporosis prevention study.

Maturitas2000;36:181-93.

45 Nagel HTC, Vanderbussche FPHA, Keirse MJNC, Oepkes D, Oosterwijk JC, Beverstock G, et al. Amniocentesis before 14 completed weeks as an alternative to transabdominal chorionic villus sampling: a controlled trial with infant follow-up.Prenat Diagn 1998;18:465-75.

46 Nicolaides K, Brizot MdL, Patel F, Snijders R. Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10-13 weeks’ gestation. Lancet 1994;344:435-9.

47 Paradise JL, Bluestone CD, Bachman RZ, Colborn DK, Bernard BS, Taylor FH, et al.

Efficacy of tonsillectomy for recurrent throat infection in severely affected children.

Results of parallel randomized clinical trials.N Engl J Med1984;310:674-83.

48 Paradise JL, Bluestone CD, Rogers KD, Taylor FH, Colborn DK, Bachman RZ, et al.

Efficacy of adenoidectomy for recurrent otitis media in children previously treated with tympanostomy-tube replacement. Results of parallel randomized and nonrandomized trials.JAMA1990;263:2066-73.

What is already known on this topic

Some people believe that participation in a randomised controlled trial (RCT) increases a patient’s risk of a bad outcome

Some people claim that the results of RCTs are not applicable to usual clinical practice

What this study adds

Participants in RCTs had similar outcomes to comparable patients who received the same or similar treatment outside the trial

The results of RCTs are therefore applicable to comparable patients in usual clinical practice

Papers

(8)

49 Playforth MJ, Smith GMR, Evans M, Pollock AV. Antimicrobial bowel preparation. Oral, parenteral, or both?Dis Colon Rectum1988;31:90-3.

50 Rigg JRA, Jamrozik K, Myles PS, Silbert B, Peyton P, Parsons RW, et al. Design of the multicenter Australian study of epidural anesthesia and analgesia in major surgery: the MASTER trial.Control Clin Trial2000;21:244-56.

51 Rosen MA, Roizen MF, Eger II EI, Glass RH, Martin M, Dandekar PV, et al. The effect of nitrous oxide on in vitro fertilization success rate.Anesthesiol1987;67:42-4.

52 Rovers MM, Straatman H, Ingels K, van der Wilt GJ, van den Broek P, Zielhuis GA.

Generalizability of trial results based on randomized versus nonrandomized allocation of OME infants to ventilation tubes or watchful waiting.J Clin Epidemiol2001;54:789- 94.

53 Schmoor C, Oleschewski M, Schumacher M. Randomized and non-randomized patients in clinical trials: experiences with comprehensive cohort studies.Stat Med 1996;15:263-71.

54 Strandberg TE, Salomaa VV, Vanhanen HT, Naukkarin VA, Sarna AJ, Miettinen TA.

Mortality in participants and non-participants of a multifactorial prevention study of cardiovascular diseases: a 28 year follow up of the Helsinki businessmen study.Br Heart J1995;74:449-54.

55 Sullivan MP, Fuller LM, Chen T, Fisher R, Fryer C, Gehan E, et al. Intergroup Hodgkin’s disease in children study of stages I and II: a preliminary report.Cancer Treat Rep 1982;66:937-47.

56 Urban P, Stauffer J-C, Bleed D, Khatchatrian N, Amann W, Bertel O, et al. A randomized evaluation of early revascularization to treat shock complicating acute myocardial inf- arction.Eur Heart J1999;20:1030-8.

57 Villamaria FJ, Baisden CE, Hillis A, Rajab MH, Rinaldi PA. Forced-air warming is no more effective than conventional methods for raising postoperative core temperature after cardiac surgery.J Cardiothorac Vasc Anesth1997;11:708-11.

58 Wetzner SM, Vincent ME, Robbins AH. Ceruletide-assisted cholecystography: a clinical assessment.Radiology1979;131:23-6.

59 Wikdahl AM, Granbom L, Stegmayr BG. CAPD bag changing with integrated discon- nect system gives lower incidence of peritonitis than with UV-box system.Adv Perit Dial 1992;8:276-80.

60 Williford WO, Krol WF, Buzby GP. Comparison of eligible randomized patients with two groups of ineligible patients: can the results of the VA total parenteral nutrition clinical trial be generalized?J Clin Epidemiol1993;46:1025-34.

61 Yamamoto H, Hughes RW, Schroeder KW, Viggiano TR, DiMagno EP. Treatment of benign esophageal stricture by Eder-Puestow or balloon dilators: a comparison between randomized and prospective nonrandomized trials. Mayo Clinic Proc 1992;67:228-36.

62 Yersin B, Besson J, Duc-Mingot S, Burnand B. Screening and referral of alcoholic patients in a general hospital. A clinical trial.Eur Addict Res1996;2:94-101.

63 Young JDH, MacEwen CJ, Ogston SA. Congenital nasolacrimal duct obstruction in the second year of life: a multicentre trial of management.Eye1996;10:485-91.

(Accepted 31 March 2005) bmj.com 2005;330:1175

Norwegian Health Services Research Centre, PO Box 7004, 0130 Oslo, Norway Gunn Elisabeth Vistsenior researcher

Doris Tove Kristoffersenstatistician Andrew David Oxmanresearcher

National Research Centre for Rehabilitation in Rheumatology, Diakonhjemmet Hospital, Oslo

Kåre Birger Hagensenior researcher

Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre, Hamilton, Ontario, Canada

P J Devereauxsenior researcher Dianne Bryantsenior researcher

Correspondence to: G E Vist gev@nhsrc.no