Doing meta-analysis in research: A systematic approach

Vivek Jain¹ , Rinku Sharma¹ , Saudan Singh²

¹ Department of Community Medicine, Vardhman Mahavir Medical College, New Delhi, India
² Director General (Medical Education), Government of Uttar Pradesh, India

Correspondence Address:
Vivek Jain
Department of Community Medicine, Vardhman Mahavir Medical College, New Delhi 110 029
India

Introduction

′Data′ consist of discrete observations of attributes or events that carry little meaning when considered alone. Analysis of data is a key component of any research; it can be of following types: ^[1]

• Primary analysis: Is the analysis of data from a single study to test the hypotheses originally formulated.
• Secondary analysis: Is the re-analysis of data from a single study to test new hypotheses or to apply more appropriate statistical procedures to test the original hypotheses.
• Meta-analysis: Is the application of statistical procedures to examine tests of a common hypothesis from more than one study.

Data collected needs to be transformed into ′information′, by reducing them, summarizing them and adjusting them for variations, such as age and sex composition of the population, so that comparisons over time and place are possible. Data that are not transformed into information and information that is not transformed into ′intelligence′ to guide decision-makers, policy makers, planners, administrators and health care personnel themselves are of little value. ^[1] To summarize information, the traditional approach consists of utilizing expert opinions, consensus statements (group expert opinion) and writing narrative review articles.

′Narrative reviews′ are summaries of qualitative research which address a broad question comprehensively. There are several limitations of narrative reviews, namely: ^[2]

• Lack of explicit descriptions of systematic methods: In order to ensure broader coverage, methods are less explicit, less clearly stated and nonreproducible by interested readers.
• Evidence can be tenuous, incomplete, or biased: Narrative reviews generally are comprehensive, covering a wide range of issues within a topic making it too broad. There is a criterion-based selection of relevant evidence and inferences are too evidence-based, a source of inadvertent incompleteness and potential selection bias.
• Readers may not be aware of selection and assessment procedure methods and thus could not make judgments about author′s choices: Typical narrative reviews do not explain how decisions were made about relevance and validity of included studies.
• Rely on statistical significance for evaluation and comparison: Simply by increasing the sample size, even a weak effect can be made to appear stronger.
• Many research literatures have grown too large for a human to accurately synthesize without the aid of statistical inference: Availability of internet-based search in addition to traditional methods of medical literature review has made the evidence too voluminous to handle.

To overcome these limitations, the new approach of systematic reviews is utilized which is an explicit quantitative synthesis of all the available evidence in literature.

′Systematic review′, regarded as the strongest form of medical evidence, is an exhaustive exploration, critical evaluation and synthesis of all the relevant studies on a specific topic. It can be a quantitative or qualitative review of available data. It employs an objective and transparent approach for research synthesis, with the aim of minimizing bias and limiting random error, thereby improving reliability and accuracy. A systematic review may include meta-analysis or statistical summary of individual studies.

What is Meta-Analysis?

The term ′Meta′ implies something occurring later, more comprehensive, and is often used to name a discipline designated to deal critically with the original one. ^[3] Meta-analysis is an objective, optional component of systematic review that draw on statistical methods to combine and summarize the results of several studies; also known as ′pooled analyses′. ^[4] It is a quantitative synthesis of all the unbiased evidence. A good meta-analysis aim for complete coverage of all germane studies, look for the presence of heterogeneity, and explore the robustness of key findings.

Purpose of Doing Meta-Analysis

• To summarize the large volume of data for easy comprehension.
• To establish the presence, and determine the magnitude, of an effect.
• To increase the power and precision of studies.
• To resolve conflicts among different reports.
• To document need for further trial(s), if any.
• To shed light into areas with insufficient research.
• To investigate variations, if any.

One of the essential purpose of doing meta-analysis is to save time and expense. By performing quantitative synthesis of all available evidence over a short time, it purges away the need to conduct fresh epidemiological studies, especially cohort studies which are time consuming and relatively expensive due to extensive follow-up of study subjects.

Need of Meta-Analysis in Dermatology

Dermatology is currently riding an enormous wave of transformation in the field of medicine and a lot of research is going on, especially, in the sub-fields of cosmetic dermatology, dermatopathology, immunodermatology, pediatric dermatology and teledermatology. A lot of documented literature on these researches is also available in various journals and web-based databases. ^[5] Meta-analysis comes across as a potential research tool to help synthesize the results of various quantitative studies on a dermatological issue. Not only it attempts to resolve the differences between conflicting evidence, but at the same time, it also paves way for identifying further areas of research. Following are a few of the meta-analyses which have been useful to researchers, scientists, doctors and other medical professionals in the field of dermatology in the recent past:

• Efficacy of topical pimecrolimus and tacrolimus in atopic dermatitis ^[6]
• Role of wound care in diabetic neuropathic foot ulcers ^[7]
• Toxicities of sorafenib ^[8]
• Comparison of efficacy of continuous terbinafine with intermittent itraconazole for toenail onychomycosis ^[9]
• Griseofulvin efficacy in the treatment of tinea capitis ^[10]

Steps of Doing Meta-Analysis

One of the reasons that researchers developed meta-analysis is to provide a way of applying the scientific methods used in primary research to the process of reviewing. The steps to performing a meta-analysis therefore have some fairly direct parallels to the steps of primary research. These steps are adapted, in part, from a compilation by Jamie DeCoster, Department of Psychology, University of Alabama, USA. ^[11]

Step 1. Make a hypothesis and define the domain of research: The initial step is to identify the problem. Additional potential confounders of the results should also be identified at this time, although other factors may be recognized during the evaluation or data collection phase of the meta-analysis. ^[12] The protocol for meta-analysis, detailing the steps that follow, should be written down before the review is actually begun.

While starting a meta-analysis, one should always be concerned about two important issues. First, the chosen subject must have the potential for constructing a discrete contribution to the theoretical concepts in field of medicine. Second, but equally important prerequisite is the availability of sufficient quantifiable literature for analysis. A good meta-analysis can also help public health policy makers, usually to determine whether an intervention had an impact (and its magnitude) on health practices. ^[13]

Step 2. Establish criteria for including studies in the review: Define a specific set of inclusion and exclusion criteria that studies must meet to be included in the analysis. Inclusion/exclusion of studies is based on several criteria, namely, published versus unpublished study, time period covered in the review, design of the study, operational definitions of the variables, quality of a study, etc.

If absolutely necessary then these criteria may be revised even at a later stage; but if the coding has started, then one should remember to recheck studies which have been already completed. Protocol changes do have the potential to introduce bias and should only be done after careful consideration. Any changes to the protocol have to be explicitly described in a revised protocol or in the text of meta-analysis.

Step 3. Literature search: Once the peripheral limits of meta-analysis are determined, trace all studies that suit within. The steps to a comprehensive literature search are:

• Search the literature meticulously to locate all possible candidate studies for analysis using reasonably candid guidelines.
• As a best practice, prepare an exhaustive ′Master candidate list′, a list of the studies that turned up in initial search of literature. It helps in avoiding the repetition of studies.
• Gain access to each of these studies for further examination.

It is proposed that the key to quality in meta-analysis lies in the results being transparent and checkable. ^[14] To ensure replicability and transparency, a systematic approach to deal with missing data is desirable. ^[15] There are several sources to help access the medical literature:

• Computerized database indices: Several web-based medical literature databases are available at World Wide Web, either on a free-for-all basis or on a payment basis. PubMed (www.ncbi.nlm.nih.gov/pubmed/), IndMED database (www.medind.nic.in), MEDLINE (www.medline.cos.com), Search Medica (www.searchmedica.com), Google Scholar (www.scholar.google.com) and Google and Yahoo search engines (www.google.com, www.search.yahoo.com) are quite useful for access to medical literature of specific interest. One should make himself/herself well verse of contents, methods of search and limitations of these databases adequately, so as to reap maximum benefits. For dermatology, one should also seek specific databases, namely, DermIs, Genamics Journal Seek and DermNet. ^[5] Details on various available medical literature databases on internet can be found elsewhere. ^{[5],[16],[17],[18]}
• Ancestor and descendent search: Always examine the references of articles which have been decided to be included in meta-analysis to see if they contain any relevant studies of which the researcher is unaware. Also, if a small number of important studies that were performed at early dates can be located, citation indices can be used to locate later articles that cite them in their references.
• Research registers.
• Theses and dissertations.
• Letters to active researchers.
• Programs from professional meetings.
• Personal contact and peer consultation.
• Public nonprofit organizations sponsoring various studies/trials.

While performing a comprehensive search of the literature, it is advisable to make use of a spreadsheet or a database program. For each study in the master candidate list, record information on a brief reference to the study, the journal or book number (if any), its current retrieval status, included or excluded from the analysis, criterion used for exclusion. To provide an accurate estimate of an effect it is important to include unpublished articles and foreign studies for analysis.

Even while writing this article, a comprehensive internet based medical literature search was carried out on various databases, especially PubMed and Google Scholar. Articles with high number of citations (as depicted in Google Scholar) and from varying fields in medicine were chosen for reference. Referenced articles were published in varied journals over a span of 20 years, thereby reflecting upon the transformation in types of review studies undertaken in the field of medicine. An attempt was also made to ensure inclusion of meta-analyses of various types of epidemiological study designs, namely descriptive, analytical and experimental designs.

Step 4. Select the final set of studies: To filter out and select the relevant studies only is the ′most fundamental challenge in a meta-analysis′. ^[19] Examine each of the studies on this list and determine whether they meet the criteria for inclusion. It is absolutely acceptable to exclude any number of studies from the initial list.

Characteristics of study quality should always be kept in mind to help reduce various types of biases. It should include:

• Design of the study: Meta-analysis, pooled analysis and randomized controlled trials are considered study designs providing strongest form of epidemiological evidence.
• Sample size: Higher the sample size, more valid the results are considered.
• Systematic literature search.
• Loss to follow-up and intention to treat analysis.

Step 5. Extract data on variables of interest, sample sizes, effect sizes, reliability of measurement and other characteristics of each study: It is obligatory that more than one reviewer extracts the data, using the predetermined forms; latter being based on patient characteristics, study design and methods, study results and methodology quality. Structured forms must be designed to capture relevant information in a concise and focussed fashion. ′Kappa statistic′ may be used to determine the level of agreement but all the differences must be resolved by consensus.

Step 6. Code each study for characteristics and effect size: Coding is the formal process of entering data in predesigned formats. Once the sample of studies has been collected, code their characteristics, viz. study ID, references (long and short), all moderating variables (to be examined), information about the overall design, and information on calculation of effect size and calculate the effect sizes.

The steps of a good coding procedure are:

• Decide the characteristics to be coded.
• Specify the unit of measurement on the basis of scale of measurement.
• Preferably a ′Code book′ must be prepared containing instructions on how to code each characteristic, citing specific relevant examples as per requirement.
• Piloting of coding scheme should be done and coders must now be trained.
• Once the method is established, start coding the studies, with the coders working independently.
• Calculate the reliability (interobserver agreement) of the coding for each item in the scheme, as a measure of the consistency; although more important will be to make sure that everyone agrees on what is to be coded. When reporting the reliability of the coding, use Cronbach′s α (for continuous variables) or Cohen′s κ (for categorical variables) for intraclass correlation

Step 7. Calculating effect sizes and interpretations: ′Effect size′ is a measure of the strength of the relationship between two variables. Effect size measures are the ′common currency of meta-analysis studies′ that summarize the findings from a specific area of research. ^[20] Effect sizes can be calculated and interpreted using ′Cohen′s d statistic′ (for categorical variable) or ′Correlation coefficient r′ (for continuous variable).

An integral step to meta-analyzing a sample of studies is to describe the general distribution of effect sizes. A useful way to describe a distribution is to report its general shape and any significant deviations. Mean and variance of effect sizes across studies is determined; the mean effect size is weighted by sample size.

Two models can be utilised to predict the combined effect sizes, namely

• Test of homogeneity - ′Fixed effects model′ (FEM): Is used for analysing within-study variance and detecting random sampling errors.
• Test of heterogeneity - ′Random effects model′ (REM): Is used for analysing between-study variance and detecting differences due to differing populations.

Where as FEM is an easy-to-apply model, comparatively REM is believed to be a better model. REMs assume that the studies observed are a random sample, allowing the generalization of population from which the sample was drawn. This allows a great deal of more freedom to apply the inferences to new situations.

′Fail Safe N′ statistics should be calculated to provide the number of nonsignificant studies necessary to reduce the effect size to nonsignificant value. This helps the researcher to gain more confidence in the stability of results obtained. ^[21]

Step 8. Select potential moderators and examine their relationships: A meta-analysis is really just an observational study. A ′moderator′ is a third variable that affects the relationship between the two variables (′Interaction′); it should be known to the researcher, if any. Moderation analysis involves use of linear multiple regression analysis or causal modeling.

Step 9. Report writing for meta-analysis: A meta-analysis report must be as complete and clear as possible. State, in each section, every decision that was made that affected the analysis, and it should be described in as plain terms as possible. It must be written comprising of the following components,

Introduction

It should concretely define the topic area of your analysis and place the topic into a broader context. It ought to include:

• A generalised description of literature that is to be analyzed along with a discussion of theoretical conflicts.
• An explanatory justification on the necessity of meta-analysis for the given topic.
• Any terminology/technical jargon used in the paper.
• Details about the process of analysis of literature.
• A precise definition of the effect being examined, a theoretical description of the boundaries of the analysis, a description of any significant subgroups of studies found in the literature and the theoretical background behind any statistical models decided to be tested.

Methods

Describe search procedures in such a detailed way that other researchers could replicate the work. It should include description of search procedure, retrieval of studies, inclusion and exclusion criteria, coding and calculation of effect sizes.

Results

In the results section, describe the distribution of effect sizes and present moderator analyses that have been decided to be performed.

• To describe the distribution of your effect sizes, present a forest plot of the effect sizes, a discussion of possible outliers and the descriptive statistics.′Forest plot′ is a graphical display that shows the strength of the evidence in quantitative scientific studies. ^[22]
• ′Funnel plot′ is a useful graph designed to check the existence of publication bias in meta-analyses. ^[23] It is a scatter-plot of treatment effect (x-axis) against a measure of sample size (y-axis), i.e., effect vs.precision.

Discussion

• Present references to other established effect sizes and ′file-drawer statistic′(which refer to Publication bias) and other statistics designed to provide intuitive meaning to affect sizes (to help audience interpret the mean effect size).
• Attempt to provide an explanation for any significant moderators revealed by the analyses.
• Discuss the diversity of the studies in the sample.
• Make theoretical inferences based on the results and evaluate its implications for the major theoretical perspectives.
• Mention any feature(s) of the analysis that might limit the generalizability of the results.
• Conclude with specific recommendations for the direction of future research.

As far as reporting meta-analyses are concerned, the QUOROM (quality of reporting of meta-analyses) group recommended a statement, a checklist, and a flow diagram. The checklist describes their preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. A flow diagram provides information about both the numbers of randomized controlled trials identified, included, and excluded and the reasons for exclusion of trials. ^[24]

Step 10. Validity of meta-analysis: A good meta-analysis uses appropriate methods of data collection and analysis (possesses internal validity), properly represents the literature being analyzed (possesses external validity), and provides a distinct theoretical contribution to the literature.

Internal validity: A meta-analysis can never be more valid than the primary studies that it is aggregating.

• Enough studies must be included to provide power for its test. To minimise Type II statistical errors, it is more pertinent to include studies with more number of patients.
• A meta-analysis should also report the relationships between moderators (if any).
• Critically examine all results involving correlated moderators to see if there is a logical reason to doubt the interpretation of the results.
• All meta-analyses should have at least two authors to ensure coding reliability.
• If there are a large number of effect sizes, the authors should report their results both including and excluding these values from their analyses.

External validity: The most important factor affecting the external validity of a meta-analysis is the representativeness of the sample of studies. Ideally the sample of a meta-analysis should contain every study that has been conducted bearing on the topic of interest.

Theoretical contribution: A meta-analysis should not simply be a summary of a literature, but should provide a theoretical interpretation and integration. In general, the more a meta-analysis provides beyond its statistical calculations the more valuable its scientific contribution. Miller and Pollock have divided meta-analyses into three categories based on their purpose and the type of information that they provide.

• ′Type A analyses′ summarize the strength of an effect in a literature.
• ′Type B analyses′ attempts to examine what variables moderate the strength of an effect.
• ′Type C analyses′ attempt to use meta-analysis to provide new evidence in relation to a theory.

Type A analyses can be seen to make the smallest theoretical contribution, followed by Type B and then Type C. While this is only a gross division (a well-conducted Type B analysis is definitely more valuable than a poorly-conducted Type C analysis, for example), it serves to highlight the fact a good meta-analysis provides more than a statistical summary of the literature.

A good meta-analysis does not simply report main effect and moderator tests. It also puts effort in interpreting these findings, and presents how they are consistent or inconsistent with the major theories in the existing literature. Meta-analyses can greatly aid a literature by providing a retrospective summary of what can be found in the existing literature. This should be followed by suggestions of what areas within the literature still need development. A good meta-analysis encourages rather than impedes future investigations.

Strengths of Meta-Analysis

• ′Provides a point estimate of an effect size′ that is much closer to the population effect size than any single study can provide. It tends to increase power so as to identify an effect, especially which may have been missed in an individual study.
• Commonly ′report confidence intervals around effect sizes′ so one can know if his/her findings are strange.
• May help to ′Identify few gaps′ in a particular field.

Limitations of Meta-Analysis

• ′GIGO principle - Garbage-in, garbage-out procedure′: Results can only be as good as the original data is valid. ^[25]
• ′Apples and oranges effect′: Tendency to average/mash together disparate effects which may have resulted from the heterogeneity of the studies. ^[26]
• ′File drawer effect′: Publication bias is a tendency of researchers, editors, publishers to handle studies with positive (significant) results differently from those with negative or inconclusive results. This leads to a significant bias in overall published literature. ^[21]
• Meta-analysis ′relies on shared subjectivity′ rather than objectivity due to subjective decisions being undertaken at various stages.

Modern meta-analysis does more than just combine the effect sizes of a set of studies. It can test if the studies′ outcomes show more variation than the variation that is expected because of sampling difference among research participants. If that is the case, study characteristics such as measuring instrument used, population sampled, or aspects of the studies′ design are coded; these characteristics are then used as predictor variables to analyze the excess variation in the effect sizes. Some methodological weaknesses in studies can also be corrected statistically.

Controversies in Meta-Analysis

Despite being strongest form of epidemiological evidence, even meta-analysis has been found to be associated with quite a few controversies.

• There has recently been disagreement in the literature on the results and interpretation of meta-analyses in few trials of, both in terms of the quantification of the effect on a disease and as regards the evidence of any adverse effect on other causes of death. Fixed effect approach to estimation relies on the implausible assumption of homogeneity of treatment effects across the trials, and is therefore likely to yield confidence intervals which are too narrow and inflexible conclusions. Conventional random effects method relies on its own set of unrealistic assumptions, and cannot be regarded as a robust solution to the problem of statistical heterogeneity. ^[27]
• Meta-analysis fails to recognise substantial clinical differences that could lead to some heterogeneity in the observed results. ^[28]
• Implementation problems in meta-analysis: It is quite difficult to explain the disparate conclusions of reviews at each step in a meta-analysis. There is always a need to specify the techniques for inclusion criteria, guidelines for the systematic summary of study features, a call for the analysis of statistical power and the reduction of Type II errors, and a discussion of the analytical problems posed by the presence of between- and within-study findings. A good meta-analysis has to ensure and reflect inclusion of issues of quality assessment, survival analysis, rare events, and sensitivity and specificity. ^[29]
• Meta-analytic predictors are categorical and non-randomly distributed affecting the statistical inferences. Meta-analysis is mostly nonexperimental and yet tries to draw a causal inference. ^[30]

Researchers are in constant look-out for ways to minimize these controversial issues in meta-analysis.

Meta-analysis has led to a paradigm shift of emphasis, from single studies to multiple studies, in research over the last few decades. It has emphasized the practical importance of the effect size instead of the statistical significance of individual studies. This shift in thinking has been termed ′Meta-analytical thinking′.

1.	Glass GV. Primary, secondary, and meta-analysis of research. Educ Res 1976;5:3-8. [Google Scholar]
2.	Collins JA, Fauser BC. Balancing the strengths of systematic and narrative reviews. Hum Reprod Update 2005;11:103-4. [Google Scholar]
3.	Egger M, Smith GD. Meta-analysis: Potentials and promise. BMJ 1997;315:1371-4. [Google Scholar]
4.	Blettner M, Sauerbrei W, Schlehofer B, Scheuchenpflug T, Friedenreich C. Traditional reviews, meta-analyses and pooled analyses in epidemiology. Int J Epidemiol 1999;28:1-9. [Google Scholar]
5.	Jain V, Raut DK. Medical literature search dot com. Indian J Dermatol Venereol Leprol 2011;77:135-40. [Google Scholar]
6.	Ashcroft DM, Dimmock P, Garside R, Stein K, Williams HC. Efficacy and tolerability of topical pimecrolimus and tacrolimus in the treatment of atopic dermatitis: Meta-analysis of randomised controlled trials. BMJ 2005;330:516. [Google Scholar]
7.	Margolis DJ, Kantor J, Berlin JA. Healing of diabetic neuropathic foot ulcers receiving standard treatment. Diabetes Care 1999;22:692-5. [Google Scholar]
8.	Zhang L, Zhou Q, Ma L, Wu Z, Wang Y. Meta-analysis of dermatological toxicities associated with sorafenib. Clin Exp Dermatol 2011;36:344-50. [Google Scholar]
9.	Trivedi NA, Shah PC. A meta-analysis comparing efficacy of continuous terbinafine with intermittent itraconazole for toenail onychomycosis. Indian J Dermatol 2010;55:198-9. [Google Scholar]
10.	Gupta AK, Cooper EA, Bowen JE. Meta-analysis: Griseofulvin efficacy in the treatment of tinea capitis. J Drugs Dermatol 2008;7:369-72. [Google Scholar]
11.	DeCoster J. Meta-analysis Notes 1998. Available from: http://www.stat-help.com/Meta%20analysis%202009-07-31.pdf. [Last accessed on 2011 Aug 15]. [Google Scholar]
12.	Berman NG, Parker RA. Meta-analysis: Neither quick nor easy. BMC Med Res Methodol 2002;2:10. [Google Scholar]
13.	Muellerleile P, Mullen B. Sufficiency and stability of evidence for public health interventions using cumulative meta-analysis. Am J Public Health 2006;96:515-22. [Google Scholar]
14.	Senn SJ. Overstating the evidence: Double counting in meta-analysis and related problems. BMC Med Res Methodol 2009;9:10. [Google Scholar]
15.	Higgins JP, White IR, Wood AM. Imputation methods for missing outcome data in meta-analysis of clinical trials. Clin Trials 2008;5:225-39. [Google Scholar]
16.	Muin M, Fontelo P. Technical development of PubMed interact: An improved interface for MEDLINE/PubMed searches. BMC Med Inform Decis Mak 2006;6:36. [Google Scholar]
17.	Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses. FASEB J 2008;22:338-42. [Google Scholar]
18.	McEntyre J, Lipman D. PubMed: Bridging the information gap. CMAJ 2001;164:1317-9. [Google Scholar]
19.	Chalmers I, Dickersin K, Chalmers TC. Getting to grips with Archie Cochrane's agenda. BMJ 1992;305:786-8. [Google Scholar]
20.	Arnqvist G, Wooster D. Synthesizing research findings in ecology and evolution. Trends Ecol Evol. 1995;10:236-40. [Google Scholar]
21.	Rosenberg MS. The file-drawer problem revisited: A general weighted method for calculating failsafe numbers in meta-analysis. Evolution 2005;59:464-8. [Google Scholar]
22.	Lewis S, Clarke M. Forest plots: Trying to see the wood and the trees. BMJ 2001;322:1479-80. [Google Scholar]
23.	Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34. [Google Scholar]
24.	Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM Statement. Onkologie 2000;23:597-602. [Google Scholar]
25.	Rosenthal R, DiMatteo MR. Meta-analysis: Recent developments in quantitative methods for literature reviews. Annu Rev Psychol 2001;52:59-82. [Google Scholar]
26.	Sharpe D. Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. ClinPsychol Rev 1997;17:881-901. [Google Scholar]
27.	Thompson SG. Controversies in meta-analysis: The case of the trials of serum cholesterol reduction. Stat Methods Med Res 1993;2:173-92. [Google Scholar]
28.	Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated. BMJ 1994;309:1351-5. [Google Scholar]
29.	Hasselblad V, Mosteller F, Littenberg B, Chalmers TC, Hunink MG, Turner JA, et al. A survey of current problems in meta-analysis. Discussion from the agency for health care policy and research interPORT work group on literature review/meta-analysis. Med Care 1995;33:202-20. [Google Scholar]
30.	Shadish WR Jr, Sweeney RB. Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes. J Consult Clin Psychol 1991;59:883-93. [Google Scholar]