Strengths and weaknesses of meta-analyses and megatrials

Applies to both meta-analyses and megatrials
  • Can ascertain moderate but worthwhile treatment benefits (small effect on major outcomes, such as death or disablement). This characteristic is important because nowadays we can seldom expect a large treatment gain by breakthrough technology.

  • Samples and results are heterogeneous not only in meta-analysis but also in one megatrial. Ironically, however, despite cries for “tailor made” medicine, it is usually the overall results of the meta-analysis or megatrial and not post hoc subgroup results that are more generalisable.

  • Meta-analyses and megatrials tend to disagree 10–30% of the time, beyond chance.

  • Even the largest megatrials are too small—that is, they are not big enough to tell us much about subgroups. Moreover, in megatrials clinical data that would allow analysis of important subgroups are often not collected for the sake of simplicity. In meta-analyses, subgroups are either not reported or are inconsistently defined across trials.

  • The patients in a megatrial are always pathologically and prognostically heterogeneous; the average RR and NNT does not apply to anyone.

  • Cannot address questions about mechanism of actions of the intervention being studied.

Meta-analysisCombination of data from several independently performed single or multicentre trials with the purpose of assessing effects on endpoints for which the individual trials are usually non-informative due to lack of statistical power.
  • Provides the most reliable treatment estimate in the absence of a definitive trial.

  • Although quite labour intensive, less expensive to conduct than a megatrial.

  • Can be seen as exploratory and hypothesis generating for the planning of a definitive large trial.

  • Biases and flaws of individual trials are incorporated, and new sources of bias may be incorporated (publication bias, prematurely terminated studies, small studies)

  • In addition to publication bias of trials, publication bias of outcomes is huge (not all identified trials report on the same primary outcomes in the same way).

  • Harms are even less often uniformly assessed than primary endpoints, so that harm assessment is less precise than benefit assessment.

  • Different statistical techniques can result in conflicting results, based on the same data.

MegatrialVery large randomised controlled trials, usually recruiting thousands of subjects and usually multicentred. Recruitment criteria are very broad, protocols are maximally simplified, and endpoints are unambiguous, such as death. Also often referred to as a “large, simple trial”. Typical examples are seen in cardiovascular medicine.
  • Can provide accurate estimates of pragmatic effectiveness and side effects in the real world.

  • Is designed from the beginning and conducted throughout to give precise measurement of treatment effects and side effects in question.

  • Large sample size required, and hence very expensive—for example, 100 million US dollars for GUSTO-I.

  • Simplification of recruitment and data collection increases the risks of protocol deviation, poor data quality, misclassification, and non-trial use of trial treatments, all of which create a bias towards the null hypothesis.

  • The control condition is sometimes defined as “treatment as usual” but this is often not standardised.

  • Megatrials can be properly designed only after many smaller trials have clarified the characteristics of the intervention in question.