Meta-analysis in medicine

What is meta-analysis?

Meta-analysis has been defined as ′the statistical analysis of a large collection of results from individual studies for the purpose of integrating the findings′.[1] It provides a logical framework to a research review where similar measures from comparable studies are listed systematically and the available effect measures are combined where possible.[2] The New York Times (7 January 1994) in a report on the effect of aspirin therapy for the prevention of recurrence of heart attacks or strokes, provided a definition of meta-analysis: ′A meta-analysis aims at gleaning more information from existing data by pooling the results of many smaller studies and applying one or more statistical techniques. The benefits or hazards that might not be detected in small studies can be found in a meta-analysis that uses data from thousands of patients′.[3]

What is the rationale for meta-analysis?

The rationale for such reviews is grounded firmly in several premises. Firstly, decision-makers are inundated with unmanageable amounts of information.

Secondly, to access all or most of the studies on a particular topic to arrive at a well informed clinical decision is often difficult, time consuming and cost-ineffective. Meta-analysis can provide access to information from many studies with less effort and hassle. It gives us the option to read a summary prepared by others, relying on those who have already spent time, money, and energy to summarize information from multiple studies on the topic.

Thirdly, single studies rarely provide definitive answers to clinical questions. Meta-analysis of multiple studies helps establish whether scientific findings are consistent and can be generalized across populations, settings, and treatment variations, or whether findings vary by particular subsets. It enhances precision, provides robust estimates, and answers questions that single trials are under-powered or were not designed to address.[4]

Fourthly, it helps resolve controversies and conflicting reports.[5] It is not uncommon to find some studies on a particular topic (treatment effects) report the benefits of the treatment while others report its hazards.

Fifthly, explicit methods used in meta-analysis limit bias and help improve the reliability (precision) and accuracy (validity) of conclusions. It effectively provides a gain in statistical power for average estimates, assuming that computing average estimates is appropriate to the particular situation. Clearly, if data from more than one study are available and can be combined, the ′sample size′ and, thus, ′power′ increases. In clinical trials and cohort studies, it offers an opportunity to observe more events of interest in the groups followed. Thus, when incidence or mortality is rare, combined estimates are likely to be more precise. Similarly, estimates are also likely to be more precise when data from case-control studies are pooled, especially when the exposure is either rare or extremely common.[2] Without meta-analysis, promising leads or small effects can be missed and researchers can embark on studies of questions that have been already answered.

Finally, meta-analyses identify crucial areas and questions that have not been adequately addressed with past research. Thus it documents the need for a major clinical trial. Similarly, it also confirms the sufficiency of available literature on a particular topic. Thus, it helps to avoid the time and expense of conducting another clinical trial. It has been estimated that for any clinical setting which involves a 10% risk of an event, 10,000 patients need to be randomized to demonstrate a 25% reduction in the risk with a more than 90% power and a p value less than 0.01.[6] Such trials are difficult to organize, even as multi-centre ventures, and meta-analysis provide an easy way of pooling data from smaller trials.

How do we conduct meta-analysis?

The first step involved in conducting meta-analysis is to write a protocol. It elaborates on the purpose of meta-analysis, designing a research question, searching for studies, specifying study selection (inclusion and exclusion) and appraisal criteria, deciding data extraction procedures (including statistical reanalysis), assessing combinability of studies, selecting an analytical strategy (use of models and sensitivity analysis), anticipating systematic errors (biases) and limitations, and presenting and disseminating results of the meta-analysis.

A credible meta-analysis should be based on a properly chosen, well-formulated, and answerable question.[7] The question guides the review by defining which studies will be included, what the search strategy to identify the relevant primary studies should be, and which data need to be extracted from each study. Ask a poor question and you will get a poor review. For example, ′Are anticoagulant agents useful in patients who have had stroke?′ is a poorly formulated question. A better question would be: Do oral anticoagulants reduce the incidence of recurrent strokes in patients with acute ischemic stroke compared with no treatment (no anticoagulation)? A clear question also helps the reader rapidly assess whether the review is relevant to his clinical practice.[7]

The next important step in conducting meta-analyses is locating relevant studies. As many relevant primary studies as possible must be gathered to minimize random and systematic error. Several complementary strategies (electronic databases, manual searching of journals, conference proceedings, and books, reference lists, existing study registries, personal contact with researchers, etc.) can identify relevant studies. Whichever methods are used, they must be reported in sufficient detail to allow replication. Publication bias represents one of the most important conceptual and methodological problems of meta-analysis. It arises from the fact that clinically and statistically significant positive results are much more likely to be published and presented at scientific meetings than negative results.[8] The subset of studies included in a meta-analysis would thus not be representative. In order to minimize this bias, the meta-analyst should make every effort to ensure that no important published paper or unpublished report has been missed.[9]

The next steps are selecting studies for inclusion and appraising them. In selecting studies, reviewers judge the relevance of the studies to the review question. In appraising studies, reviewers judge numerous features of design and analysis. Methodical, impartial and reliable strategies are necessary for these two tasks because meta-analyses are retrospective exercises and are therefore susceptible to both random and systematic sampling errors.[10] For example, a single trial of risperidone for chronic schizophrenia was reported in seven different publications with different authorship.[11] The danger is that data from the same patient may be analyzed more than once, leading to biased estimates.

Then follows data extraction from selected studies. To quantify and summarize effects, one needs to specify how effects will be measured. Since different researchers use different effect measures to report their findings, it becomes difficult to decide which effect measure should be used for summarization. Extracting an estimate from a published report involves no more than copying it if the report gives the desired estimate and its standard error estimate. If not, then one needs to re-analyze the statistics from primary studies by constructing. Nevertheless, even these crude methods require certain minimum information, and some reports may have to be excluded due to inadequate data presentation. In such cases, the authors can be contacted for additional information.[12]

Meta-analysis uses statistical methods to combine the effect measures extracted from primary studies. Although the statistical methods involved appear to be mathematically complex, their purpose is simple. They attempt to answer four basic questions.[13] Are the results of the different studies similar? To the extent that they are similar, what is the best overall estimate? How precise and robust is this estimate? Finally, can dissimilarities be explained?

Are the results of different studies similar (homogeneous)? To answer this, one must calculate the statistical diversity (heterogeneity) of the treatment effect that exists among the different sets of data. This heterogeneity can be due to two causes. Firstly, even if the true effect was the same in each study, the results of different studies would be expected to vary randomly around the true common fixed effect. This diversity is called the within-study variance. Secondly, each study may have been drawn from a different population. Therefore, even if each study enrolled a large patient sample, the treatment effect would be expected to differ. These differences are called the between-study variation.[13] The test most commonly used to assess the statistical significance of between-study heterogeneity is based on the chi-square distribution.[14]

To the extent that data are similar (homogeneous), what is their best summary point estimate, and how precise is this estimate? This step generally involves combining the results of different studies into an overall (summary) estimate. In this process each study is given a weight according to the precision of its results. The rationale is that studies with narrow confidence intervals should be weighted more heavily than studies with a greater uncertainty. When the combined trials are a homogeneous set a fixed-effects model is appropriate.[15] When heterogeneity is detected, use the random-effects models.[4],[13] Sensitivity analyses are then used for assessing the robustness of the overall point estimate.[4],[12],[13] Additionally, heterogeneity if present, should not simply be ignored after a statistical test is applied; rather, it should be scrutinized and explained.

The next important issue involved in conducting a meta-analysis is addressing biases and limitations, such as publication bias, selection bias, data extraction bias, and aggregation bias (ecologic bias). Other controversies in meta-analysis include the use of quality scores, inclusion of unpublished studies and abstracts, and cross-design synthesis, which should be elaborated in the discussion of meta-analysis.[16]

Finally, the results of meta-analyses are typically presented in a graphic form [Figure - 1][17] that shows the point estimates and their confidence intervals. This presentation aims to convey an impression of the results of the individual studies, the extent of heterogeneity, and to report the pooled estimates.

What is the role of the Cochrane Collaboration in all this?

The Cochrane Collaboration is an international organization that aims to help people make well informed decisions about health care by preparing, maintaining, and ensuring the accessibility of rigorous, systematic, and up-to-date reviews (and where possible, meta-analyses) of the benefits and risks of health care interventions.[18] The main product of the Cochrane Collaboration is the Cochrane Library, an electronic library that contains a handbook on how to conduct systematic reviews, and four databases, The Cochrane Database of Systematic Reviews (CDSR), The Database of Abstracts of Reviews of Effectiveness (DARE), The Cochrane Controlled Trials Register (CCTR), and The Cochrane Review Methodology Database (CRMD).[19]

What is the contribution of meta-analysis to dermatology?

A meta-analysis to investigate the cost-effectiveness of topical prednicarbate 0.25% and fluocortin 0.75% in the treatment of inflammatory dermatoses, such as eczema, found that the effectiveness was 84.9% for prednicarbate and 69.7% for fluocortin, while the frequency of adverse effects was 3.5% for prednicarbate and 4.9% fluocortin.[20] Prednicarbate had a favorable cost-effectiveness ratio as compared to fluocortin, for the treatment of eczema.

A meta-analysis of 18 case-control studies published between 1977 and 1996 found no evidence for an etiological role of oral contraceptives use in the development of cutaneous malignant melanoma.[21]

A meta-analysis that included 25 case-control studies concluded that evidence to support the association between intermittent sunlight exposure and melanoma risk was still far from complete.[22] A meta-analysis of 9067 patients from 11 case-control studies inferred that the available epidemiological data do not support the existence of a relationship between topical sunscreen use and increased risk of cutaneous malignant melanoma.[23]

A systematic review of topical treatments for fungal infections of the skin and nails of the feet found little evidence to differentiate between three popular over- the-counter topical treatments (azoles, tolnaftate and undecenoic acid) for fungal skin infections.[24] It concluded that the evidence about the efficacy of topical treatments for nail infections is very sparse. Rigorous research is overdue.

There are more examples of the use of meta-analysis for resolving conflicting reports in dermatology. Historically, randomized controlled trials are considered gold standard study designs, but if they report discordant results and generate controversies, then the answer is meta-analysis.