Quality of reporting and of methodology of studies on interventions for trophic ulcers in leprosy: A systematic review

Introduction

Leprosy is a chronic infectious disease caused by the bacterium, Mycobacterium leprae . More than three million individuals currently have, or are disabled by, leprosy worldwide. Although the disease can be treated effectively, it is associated with irreversible peripheral neurological damages with subsequent sensory loss in the skin. As a consequence, people with leprosy are at risk of having injuries to their limbs and joints, such as thickened and cracked skin or ulceration.

In the process of conducting a systematic review on interventions for skin lesions due to neuritis in leprosy, we assessed several primary papers for inclusion. The complete review is published in the Cochrane Library. ^[1] A systematic review is an overview of primary studies that contains an explicit statement of objectives, materials, and methods, and which is conducted according to explicit and reproducible methodology. ^[2] It makes searching for and reading of multitude individual studies redundant. ^[3],[4] Naturally, the review may reveal that there exist no reliable studies or no studies at all, implying that there is a knowledge gap. The Cochrane Handbook for Systematic Reviews of Interventions offers guidance to the process of conducting systematic reviews on interventions in health care. ^[5]

To be able to draw conclusions on the effect of a treatment with some confidence, it is of vital importance that the studies included in a systematic review are of a proper design and quality. To evaluate this, a checklist of quality criteria is designed and used to assess the study reports. Several such checklists have been made, but there is as yet no gold standard. ^[5] In the Handbook it is emphasised that simple approaches for assessing internal validity should be used. However, a checklist is not of much help if the studies are inadequately reported. A poorly reported study makes it very difficult to assess its true quality.

The assessment process we undertook to make the systematic review, revealed a number of shortcomings in the reporting and methodological quality of studies. In this article we give a short account of how we identified, selected and assessed the papers and what inadequacies we found.

Methods

Literature searchesWith support from the Cochrane Skin Group we inspected the Skin Group Specialised Register and searched the Cochrane Central Register of Controlled Trials (CENTRAL) for randomised controlled trials in leprosy. We made additional searches in MEDLINE, EMBASE, CINAHL AMED and LILACS and in some selected web-resources.

Selection criteria for studiesIn the protocol we stated that studies which were clearly not randomised trials would not be eligible for the review. Participants were people affected by leprosy, being or having been on multi-drug treatment, with damage to peripheral nerves. We were looking for several types of interventions, such as education, information, self-care programmes, dressings, topical agents, skin care, footwear or other interventions designed to prevent damage. Types of primary outcome measures were prevention of skin ulcers, prevention of limb deformity or healing of existing ulcers.

ChecklistsCriteria for assessment of study quality were based on the directions in the Cochrane Handbook and the recommendations given in "Methods used in Cochrane Skin Group reviews." ^[6] According to the Cochrane Handbook there are four main threats to the internal validity of a study: selection bias, performance bias, attrition bias and detection bias. All trials were assessed for whether there existed factors that could lead to any of these biases. The items we considered were: proper method of random sequence generation and concealment of allocation (to prevent selection bias); blinding of outcome assessor (to prevent detection bias); degree of follow-up (to prevent attrition bias) and blinding of health personnel participants (to prevent performance bias - because of the difficulties of blinding the caregiver, this factor was not used in the assessment). We also assessed whether the groups in the trials were similar at baseline; whether reliable outcome measures had been used and the appropriateness of statistical analysis. When assessing the statistical analysis, we emphasised the appropriateness of choice of tests, whether the test′s assumptions had been assessed, whether the unit of analysis was the same as the unit of allocation and whether the analysis were done by intention to treat. In addition, we thought it was important to assess the adequateness of the description of the intervention, though this is an issue more related to the external than the internal validity of a study.

We did a summary assessment of quality based on the following criteria: If the first three items regarding selection, attrition or detection bias were scored as adequate and there were no important concerns regarding the other items, the protection for bias was rated as high. If one or two of the first items were scored as ′unclear′ or ′inadequate′ and there were no other important concerns, it was rated as having a moderate protection against bias. In other cases, protection was assessed to be low. When an item was not explicitly reported, we tried to deduce it from relevant information anywhere in the article.

We used the checklist for assessing methodological quality as a basis in the making of the checklist to assess the quality of reporting of items.

Data extractionThe data extraction was performed by two reviewers (LMR, LF), who independently entered data into the data extraction forms. Discrepancies were resolved by consensus. We contacted some authors for missing data, but never received any reply.

Results

The search returned 854 citations to potentially relevant trials. No ongoing trials were found. Based on their title and abstract we selected 35 papers for further assessment [Figure - 1]. These studies were retrieved in full text. Out of these studies, a total of 10 trials were identified. However, one of the studies had a selection of mixed patient groups (with leprosy, diabetes or venous ulcers) and therefore did not meet the inclusion criteria for participants. One other study had used day of attendance at the clinic as the method for group allocation. Such a method is clearly a quasi-randomised procedure and we chose to perceive this method as somewhat different in essence from alternation, which is another method labelled quasi-randomised. Therefore, this study will not be considered here. Accordingly, eight of the studies were eligible for the review, three of which had used alternate allocation. The interventions tested were dressings, topical agents and footwear. They had all used healing of ulcers as their main outcome measure. All identified studies are presented in [Table - 1].

Quality of studiesMost of the trials were judged to have poor methodological quality with a high risk of selection and detection bias [Table - 2]. For the item ′allocation concealment′ all studies were assessed as either unclear or inadequate. Only two trials seemed to have taken measures to blind outcome assessors. In half the studies the authors did not or failed to show that the groups were comparable at baseline. Two studies clearly had such losses to follow-up that it could be detrimental to the validity of the results, while another study was unclear as to how many participated in the analyses. Studies with losses 20% or below were rated as having adequate follow-up.

Three studies reported mean reduction of ulcer area as the outcome of interest. One of these had tested the reliability and accuracy by two independent researchers. One study did not report how the ulcer had been measured, and the third used a method that is perceived as valid, ^[7] but reported no quality assessments of its application. One study reported number of days to healing while the rest of the studies reported number of ulcers healed, both of which we judged as being fairly reliable measures.

Two studies had allocated patients with clearly more than one ulcer and used ulcers as the unit of analysis, while two others had potentially a unit of analysis error. As for appropriateness of statistical analyses, two studies had not done any significance tests, so they were rated as inadequate along with two other studies, three studies were assessed as unclear because it was unclear as to whether the test′s assumptions had been examined, and one study was rated as adequate.

Quality of reportingGenerally, reporting of both, methodology used in the study and of results were inadequate and rather disorganised. The most under-reported items were concealment of allocation, blinding of patients and outcome assessors, intention to treat and validation of outcomes [Table - 3]. Moreover, even if the method used to generate the allocation sequence was partly described, it was not detailed enough to judge its adequacy. For instance, stating that the patients were ′randomly allocated′ is not adequate for deciding which method was used to generate the randomisation sequence. When patients were alternatively assigned, it was not reported how the first patient was assigned. All studies failed to report on concealment of the allocation procedure, as well as who generated the allocation sequence, who enrolled the patients and who assigned the patients to the different treatment groups.

Six of the studies failed to mention whether the outcome assessor had been blinded. Follow-up, i.e. the flow of patients through the study, was poorly described and none of the studies had flow charts. Although we for the most part could deduce the number of patients from text or tables, it was not explicitly and clearly reported whether all patients still participated in the study at the end. Likewise, none of the studies reported explicitly what had been the unit of analysis, but in most cases we could deduce it.

In general, there was a lack of specification of primary and secondary outcome measures. The interventions were rather well described in all the papers, except that more detail could have been given regarding dose and strength of solution. Also descriptions of study′s objectives, settings and patients were quite acceptable. Less than a third of the papers reported having asked the participants about the acceptability of the interventions.

Discussion

This paper assessed the methodological quality and quality of reporting in studies on treatment of ulcers caused by nerve damage in leprosy. The results highlight the need for improved reporting and better quality of studies. They also reveal that there is a paucity of studies of different treatments for ulcers in leprosy. Both quality and lack of studies is a challenge for studies within ulcer treatment in general, not only within leprosy. ^[8],[9],[10] Regarding quality, this raises the question as to how improved education in research methods could be better implemented in post-graduate teaching courses in dermatology.

Awareness of weaknesses of previous studies may be used to improve planning, conduct and reporting of future clinical trials. How a study was conducted and how it was reported are two different things. It is nevertheless through the report we assess a study. It may still be possible to find small pieces of information spread around the text that eventually may answer our questions even if there may be deficiencies in the reporting. However, if the information is completely lacking, the relevant item will be scored as unclear, and thus the reporting also will influence the assessment of methodological quality. Based on the weaknesses that we identified in the reporting of studies in our review, we would like to stress the following points that could improve the quality of future study reports.

Allocation method and concealmentAt least three of the studies that we appraised stated that they had used alternation when allocating patients to groups. Treatments allocated alternately "are in principle unbiased being unrelated to patient characteristics", under the assumption that the procedure is concealed to those responsible for the assignment. ^[11] However, studies using such methods are considered to be quasi-randomised and are excluded from reviews whenever randomised controlled trials have been stated as the inclusion criterion for type of design. Although such methods are possible to conceal, they are according to the Cochrane Handbook usually not concealed. If the allocation procedure is open it is easy to manipulate by the person in charge. This may result in a selection bias which in turn will threaten the validity of the study. Even if base-line measurements which indicate that characteristics are similarly distributed are provided, one cannot know whether the same applies to unknown prognostic characteristics.

However, it is important to remember that the same requirement of concealment of the allocation schedule relates to the use of random numbers tables. ^[12] Actually, failure to conceal has been found to be more important in predicting bias than other components of allocation, such as the generation of the allocation sequence, for instance by computer, random numbers table or alternation. This implies that the randomisation sequence should be generated by an independent entity and that the implementation of the sequence should be concealed. The report should state explicitly exactly how this was done.

OutcomesRegarding reliability of outcome measures, some studies used mean percentage reduction based on the surface area of the wound. Because there are different methods of measuring the size of a wound, it should be measured by two observers and then compared. However, whether the rate of decrease of ulcer area is a good predictor of total ulcer healing is not quite established. ^[8],[9] Besides, the rate of healing may not vary linearly with the follow-up time.

According to reviewers of the effect of treatment of ulcers caused by other underlying mechanisms than leprosy, time to complete healing or proportion of wounds completely healed is the outcome of greatest interest to patients. ^[8],[9],[10] Also, the patients′ acceptability of treatment and quality of life measurements, e.g. reduction of social stigma, are often suggested as important and relevant outcomes.

BlindingIn order to prevent a biased assessment of the success of treatment it is important that the assessor is ignorant of which group the participant belongs to. ^[13] That this also applies to wound assessment is suggested by a recent study that demonstrated differences between blinded and unblinded assessors of wound progression. ^[14] Two of the three studies assessing surface area of the wound did in fact use blinded assessors. Perhaps this is not as important when using total healing as the outcome. However, if it is, it should be done and reported. Even patients, who in some studies could observe that they received different treatments, could have been blinded to the study hypothesis. Both should be explicitly reported, whether done or not.

One person - several ulcersWhen allocation to groups is done on an individual level and the participants have several ulcers, it means that one person will contribute several times in the analysis. This will represent a unit of analysis error if not corrected for in the statistical analysis. In turn, this might lead to an over-estimate of the effect because the intra-variance between healing of ulcers on the same person may be smaller than the inter-variance of healing of ulcers between individuals. One way to avoid this problem completely is to choose the worst ulcer for each person as the reference ulcer and treat, measure and analyse these. ^[15]

Consort checklistInadequate reporting of trials is an acknowledged problem in many fields, ^{[16],[17],[18]} particularly after the publishing of the CONSORT (Consolidated Standards of Reporting Trials) statement. In 1996 an international group of researchers, statisticians and editors developed a check list of 21 items and a flow diagram, describing the patient flow through the trial, to help authors improve the reporting of a study: ^[19]

"CONSORT comprises a checklist and flow diagram to help improve the quality of reports of randomized controlled trials. It offers a standard way for researchers to report trials. The checklist includes items, based on evidence, that need to be addressed in the report; the flow diagram provides readers with a clear picture of the progress of all participants in the trial, from the time they are randomized until the end of their involvement. The intent is to make the experimental process more clear, flawed or not, so that users of the data can more appropriately evaluate its validity for their purposes".

The statement was revised in 2001. ^[20] For further directions on reporting we refer to the Consort Statement website: http://www.consort-statement.org/. The guidelines should be used when writing up a paper for publication and should be consulted in advance when planning a clinical trial. This would improve both conduct and reporting of all studies.

LimitationsWe assessed the quality of reporting on the basis of rather few criteria. We could have used the whole checklist from the CONSORT statement. However, the criteria we used are those being most important when making judgements about the internal validity of a study.

Our involvement in the review is mainly from a methodological point of view as we are not involved in clinical practice and we work in a country with no leprosy cases. This might limit the validity of this present paper. However, we believe that the same quality criteria should be set for all papers reporting trials done in any setting. This would be one way forward to make sure that people with leprosy and their health care workers can make their decisions based on the best available research evidence.

Conclusion

We do acknowledge that the existing infrastructure in the leprosy field and the presumably restricted funds for treatment and research, may limit the opportunities for undertaking high quality randomised controlled trials. Nevertheless, there is an apparent need to stimulate more research and improve the methodological quality as well as the quality of reporting of trials in leprosy ulcer treatment.

The most important threat in existing studies is the threat of selection bias, which may result from failure of concealing the allocation process, and which will influence the whole study. For the reporting of future studies, journals could promote and encourage the use of the CONSORT statement checklist by expecting and requiring of authors to follow it in their reporting.^[28]