A longitudinal study of consistency in diagnostic accuracy of teledermatology tools

Introduction

The term "teledermatology" was coined by Prednia and Brown in 1994. ^[1] In 2001, Edey and Wooten ^[2] reviewed the pros and cons of both store and forward teledermatology (SAFT) and video conference (VC) tools. Huntley and Smith ^[3] in 2002 underscored the importance of internet and its role to pool experts′ opinion for difficult to manage cases. Braun et al.^[4] demonstrated telemedical wound care using the mobile phones. Besides, various traditional reviews ^[5],[6],[7] have contributed to the insights into teledermatology. In 2008, Kanthraj ^[5] proposed the classification of teledermatology practice (TP). A revised classification was presented in 2011 ^[6] to incorporate tertiary teledermatology. Teledermatologists like Emnovic et al, ^[8] Warhaw et al.^[9] and Van der Heijden et al.^[10] have systematically reviewed and summarized the application of VC, SAFT and tertiary teledermatology.

A successful implementation of TP in a given health-care setting depends on technical feasibility of a teledermatology tool (TT) and factors like patient and physician willingness and satisfaction for the technology. ^[11] The competence of TTs is demonstrated in a clinical setting by feasibility studies. The diagnostic accuracy (DA) is an outcome measure obtained from a feasibility study that evaluates a teledermatology tool when a teledermatologist diagnosis is compared with a face-to-face examination by a clinical dermatologist (gold standard) followed by the statistical analysis (kappa value) of the data.

A plethora of feasibility studies ^{[12],[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71]} have been conducted to test the competence of TTs with or without a gold standard comparison between clinical and teledermatologists. Few authors ^{[12],[13],[14],[15],[16],[17]} have compared teledermatologists with nurses, general practitioners and documented DA while others ^{[18],[19],[20],[21],[22],[23],[24]} have compared between teledermatologists with a clinical dermatologist and obtained the results. These varying outcomes have all contributed to the quandary of reliable DA value for TTs. Most notably, despite extensive and accelerated dissemination of teledermatology reports, there is no study that has examined the aggregate DA of a TT and observed its consistency over a period of time. In this milieu, a longitudinal study was undertaken for the first time to observe the DA trend of various TTs. Feasibility, the practicability of a TT in terms of both technical (technology) and clinical (diagnosis) was analyzed in this study; this is an important area of investigation because the findings could help to determine the usefulness of TP and its further implementation for community health-care program.

Methods

The steps entailed in the study were identification of studies with a single TT for general, tertiary, pediatrics and geriatrics teledermatology care, addressing dermatological conditions of general out-patient setting that are diagnosed mostly by spot examination. The purpose of this study was to compare the TT with reference to a gold standard (face-to-face examination) and we did not focus on the type of case mix involved in each study. All the 328 articles in PubMed obtained after using the search term "teledermatology" and "TP" were categorized as (a) 105 studies (b) 36 surveys (c) and 187 other than study or survey articles. The inclusion and exclusion of articles are shown in [Figure - 1]. Furthermore, in these 97 studies, 59 studies assessing the DA of single TT used for diagnostic purpose were included. The 38 studies were excluded based on the following decisive factors: (a) combination of TTs used for diagnostic purpose (13 studies) as the combination of tools would interfere in the proper assessment of a single tool. (b) Studies focusing on a single clinical entity (16 studies) (c) comparison between two TTs (4 studies) and (d) non-English articles (5 studies). Studies that employed additional or special TT like teledermoscopy were excluded.

Figure 1: Included and excluded feasibility studies on teledermatology tools (TT: Teledermatology tool, TP: Teledermatology practice, SAFT: Store and forward teledermatology, VC: Video conference, MT: Mobile teledermatology, ODF: Online discussion forum, PT: Pediatric teledermatology, GT: Geriatric teledermatology)

Gold standard diagnosis is the face-to-face consultation with histopathology confirmation. However, we considered all the teledermatology studies compared with face-to-face examination alone to be the minimum standard and were included. The DA values obtained from all the complete feasibility studies that compared the DA between the clinical dermatologist (face-to-face examination as the gold standard) and teledermatologist that evaluated a single TT were included and unified for overall analysis. The DA values obtained from studies without gold standard comparison, absence of DA comparison between teledermatologist with a clinical dermatologist, retrospective analysis were excluded. Fifty nine included studies ^{[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71]} [Table - 1] were read completely and analyzed individually based on (a) TT assessed on the number of subjects, i.e., more than 100 patients was considered as a major study while less than 100 as a small study (b) presence or absence of the gold standard comparison (face-to-face examination) between the diagnosis, i.e., primary (diagnosis) or secondary (differential diagnosis) offered by teledermatologist were compared with clinical dermatologist. Based on these criteria, the studies were graded as grade 1, the DA obtained from a prospective study after testing over a large number of patients (>100) with the gold standard comparison. A prospective small study with the gold standard comparison between a teledermatologist with a clinical dermatologist is grade 2. A prospective large study (>100 subjects) without a gold standard comparison is grade 3. A small study without a gold standard comparison is 4. A grade 5 study is a retrospective analysis of the data or a study that establishes the feasibility without documenting DA.

Table 1: Studies on store and forward teledermatology, video conference, mobile teledermatology, online discussion forum, pediatric and geriatric teledermatology

Total number of feasibility studies included to evaluate a single TT used for diagnostic purpose in general, tertiary or subspecialty care TP was 59. ^{[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45],[46],[47],[48],[49],[50],[51],[52],[53],[54],[55],[56],[57],[58],[59],[60],[61],[62],[63],[64],[65],[66],[67],[68],[69],[70],[71]} They were grouped according to the working classification of TP ^[5] [Table - 2]. Although Mobile teledermatology (MT) is a variant of SAFT and VC, MT differs in net connectivity technology and dermatology care is provided by using cell phones. Subspecialist care in pediatric and geriatric teledermatology has emerged focusing the dermatological conditions of those age groups. This is reflected in the teledermatology literature. Hence, the studies were placed as separate entities.

Table 2: Distribution of studies with gold standard comparison on various teledermatology tools and summary of the number of patients and their average diagnostic accuracy

Most of the studies 51% (33) were on SAFT ^{[13],[14],[15],[16],[17],[18],[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35],[36],[37],[38],[39],[40],[41],[42],[43],[44],[45]} for general teledermatology (Seventeen studies ^{[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35]} on SAFT with the gold standard comparison (Grade 1-2) were included, 10 studies ^{[36],[37],[38],[39],[40],[41],[42],[43],[44],[45]} without a gold standard comparison (Grade 3-5) were excluded. Furthermore, there were 6 studies on SAFT ^{[13],[14],[15],[16],[17],[18]} that were excluded as these studies had a gold standard comparison with face-to-face examination; however, the comparison was carried out by a nurse or general practitioners and not a dermatologist. This can result in variation of DA values.

In VC tool, there were 19% (10) of feasibility studies, ^{[46],[47],[48],[49],[50],[51],[52],[53],[54],[55]} eight studies ^{[46],[47],[48],[49],[50],[51],[52],[53]} with the gold standard comparison were included (Grade1-2), and two (Grade3-5) studies ^[54],[55] without a gold standard comparison were excluded. MT had four (8%) small studies ^{[56],[57],[58],[59]} (Grade 2). There were 17% (9) of feasibility studies on tertiary ^{[60],[61],[62],[63],[64],[65],[66],[67],[68]} (second opinion) teledermatology, three studies ^{[60],[61],[62]} with the gold standard comparison were included and six studies ^{[63],[64],[65],[66],[67],[68]} without a gold standard comparison were excluded for analysis on online discussion forum (ODF). The sub-specialty TP included 4% (2) and 2% (1) studies with the gold standard comparison for pediatric ^[69],[70] and geriatric ^[71] TP respectively.

Author, year of publication, number of subjects, gold standard comparison between teledermatologist versus clinical dermatologist and DA were noted from each study with respect to each TTs and analysis of DA trend over a period of time (15 years), from 1997 to 2011 was performed. They are summarized in [Table - 1].

The DA trend of a TT over a 15 year period

Grade 1 and 2 studies were included for analysis. The studies were arranged accordingly in the chronological year in which they were published. Number of studies, enrolled patients and DA were noted for a consecutive 3 year period up to 15 years (1997-2011) and the DA trend was analyzed.

Statistical analysis

Descriptive statistics, Chi-square, one-sample t-tests were employed using a statistical package (SPSS for windows version 17, SPSS Inc., Chicago, IL) to analyze the data. Non- parametric tests of Mann-Whitney U and Kruskall Wallis tests were performed.

Results

The analysis of various studies on TTs, number of studies, patients, and their average DA values are shown in [Table - 2].

Comparison of DA of SAFT, VC versus 100% accuracy as standard value

The average DA obtained from 17 studies ^{[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35]} with gold standard comparison on SAFT were 73. 35 ± 14.87 and the average value for VC with 8 studies ^{[46],[47],[48],[49],[50],[51],[52],[53]} with the gold standard comparison were 70.37 ± 7.01 [Table - 3]. Assessment with t-test (independent samples) for inter comparison between SAFT and VC with respect to the number of patients were insignificant. The DA was comparable between both the methods. There is no significant difference between DA values of SAFT and VC [Table - 3]. One sample t-test analysis with 100% accuracy as standard value revealed 28% of deficiency for SAFT (t = 7.925), P = 0.000 and 30% deficiency for VC (t = 11.955, P = 0.000).

Table 3:

Three-year consecutive DA of SAFT and VC assessed over a 15 year (1997-2011) period

Analysis of 3 year DA trend of SAFT included 17 studies ^{[19],[20],[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33],[34],[35]} with the gold standard comparison (grade 1 and 2) altogether tested on 2385 patients confirmed a consistent trend with 73.35% ± 14.87% [Table - 3]. The Kruskall Wallis test (χ² = 1.852, P = 0.763) confirmed the consistent DA values of SAFT over a 15 years period (1997-2011) [Figure - 2]. VC was evaluated by 8 studies ^{[46],[47],[48],[49],[50],[51],[52],[53]} with the gold standard comparison and 1305 patients were enrolled from 1997 to 2001. Mann-Whitney U test analyzed average DA 70.37% ± 7.01% for VC. A consecutive 3 year DA trend indicated the values were consistent. (u = 3.00, P = 0.429) [Table - 3]. There were no further studies with the gold standard comparison on VC after 2001.

Figure 2: Comparison of regular 3-year diagnostic accuracy trend of store teledermatology and forward and Videoconference over a period of 15 years (1997-2011)

Discussion

The concept of DA itself has certain pitfalls especially in the manner in which it was reported - variation in the level of training of both the referring physician and the teledermatologist, inter observer variability, and percentage agreement/kappa statistics.

There were no uniform standards followed in the methodology to conduct feasibility studies. Discrepancy to capture images, camera resolution, inter-observer variation, difference in training and expertise on the subject may explain the DA variation. The clarity of images, speed of the internet and rapidity of teleconsultation has improved compared to those used in studies of 10 years back. The wide variation in DA margin may be minimized by the following measures: (a) comparison with a gold standard face-to-face examination (b) diagnosis made by teledermatologist should be compared with a clinical dermatologist c) adherence to the standards proposed by American teledermatology association and practice guidelines ^[72] that ensure a minimum standard for TP, uniformity in the methodology with reproducible results.

Only small studies were available for MT and therefore, extensive studies are required in this field. Most of the studies on ODF ^{[60],[61],[62],[63],[64],[65],[66],[67],[68]} were retrospective analysis. However, ODF is a modification of SAFT and the principles of SAFT matches exactly with ODF. There were studies with the gold standard comparison on SAFT adjudicating that SAFT is a time-tested technology for the past 15 years with better DA as analyzed in the present study and separate studies on technical feasibility of ODF may not be required.

There are sparse studies with the gold standard comparison that encourage sub-specialty care like pediatric ^[69],[70] and geriatric ^[71] teledermatology. Research in these areas would facilitate the implementation of teledermatology program in a health-care setting. The 15 years (1997-2011) data analysis confirmed SAFT and VC were the tools evaluated by studies with the gold standard comparison on a significant number of patients with good DA. SAFT is a regularly validated tool from past one and half decades with a consistent DA when compared to VC. The DA is almost similar in both SAFT and VC; however, SAFT has a consistent DA and it is an easy, convenient and cost-effective ^{[73],[74],[75]} tool that makes it the most widely used technology.

The stable DA despite the technical advances in this field suggests dermatological conditions that can be diagnosed by face-to-face examination be able to diagnosed by teledermatology. Though, VC has a consistent and good DA, general practitioners, dermatologists and patients are required for simultaneous interaction making this sort of practice a more time consuming tool. SAFT is being used as an effective alternative to VC and dermatologists are using SAFT frequently and consistently for research and practice.

The current longitudinal study observed that feasibility studies have shown both SAFT and VC tools were adequately validated with large feasibility studies involving ample number of patients with the gold standard comparison between teledermatologist and clinical dermatologists. The DA values for both SAFT (average 73.35%) and VC (average 70.37%) are good and comparable. Hence, these two tools are feasible for TP. SAFT is simple and easy to use. However, it has a limitation of absence of patient interaction with dermatologists, which is practiced by videoconference. A combination of SAFT and VC-hybrid teledermatology can improve DA, provide better patient satisfaction and disadvantages of SAFT or VC used alone can be overcome by this combination of SAFT and VC.

A significant difference was observed when SAFT and VC were compared with face-to-face (gold standard) and when DA is assumed as 100% accuracy (standard value). Dermatology is a visual specialty and most of the dermatological conditions can be diagnosed by face-to-face consultation (considered as the gold standard) or spot diagnosis alone. However, certain conditions like pigmented skin lesions may not be diagnosed appropriately by face-to-face and require additional investigations. Therefore, those dermatological conditions that are diagnosed by face-to-face can be diagnosed by teledermatology. However, dermatological conditions with ambiguity may require any of the following two approaches (a) initial TP followed by face-to-face examination and (b) initial face-to-face examination followed by TP. Combination of both face-to-face and teledermatology in appropriate dermatological conditions could deliver quality care. ^[6] Doubtful cases can be submitted to ODF like ACAD_IADVL@yahoogroups.com - an E-mail group formed by the members′ of academic societies such as Indian association of dermatologists,′ venereologists, and leprologists to pool expert opinions rapidly and deliver dermatology care in reduced time.

Conclusion

In conclusion, the present study assessed various feasibility studies of single TT addressing dermatological conditions of general out-patient setting that are diagnosed mostly by spot examination and excluded studies addressing combination of TTs like teledermoscopy. Pigmented skin lesions like melanoma require teledermosopy for their management. The burden of pigmented skin lesions as a community health problem is negligible in India compared to the west. Therefore, this study has significant relevance to Indian context as Indian teledermatology rarely requires teledermoscopy compared to the west. The data analysis of this study suggests basic TTs like SAFT provides best DA. Health-care providers need to plan for appropriate utility of SAFT either alone or in combination with VC to implement and deliver teledermatology care in India.

Acknowledgments

Author is thankful to Dr. Lancy D′Souza, Reader in Psychology, Maharaja′s College, University of Mysore, Mysore, and Karnataka, India for statistical analysis of the data. Mr. Coimbatore Krishnan Muralidharan, Mr. Manjesh Karigowda, and Mr. Avinash Mysore for secretarial assistance involved in this project. Author is thankful to the Indian society of teledermatology (INSTED) and Special interest group (SIG) teledermatology of the Indian Association of Dermatologists, Venereologists and Leprologists (IADVL) and J. S. S. University, Mysore for their constant academic encouragement rendered in completion of this project.