This article has Open Peer Review reports available.
Implementation of blinded outcome assessment in the Effective Verruca Treatments trial (EverT) – lessons learned
© The Author(s). 2016
Received: 10 October 2015
Accepted: 8 July 2016
Published: 12 July 2016
Trials using inadequate levels of blinding may report larger effect sizes than blinded studies. It has been suggested that blinded outcome assessment in open trials may in some cases be undertaken by assessments of photographs. The aim of this paper is to explore the effect of using different methods to assess the primary outcome in the EVerT (Effective Verruca Treatments) trial. It also aims to give an overview of the experiences of using digital photographs within the trial.
We undertook a secondary analysis to explore the effect of using three different methods to assess the primary outcome in the EVerT trial: assessment of digital photographs by blinded healthcare professionals; blinded healthcare professional assessment at the recruiting site and patient self-report. The verruca clearance rates were calculated using the three different methods of assessment. A Cohen’s kappa measure of inter-rater agreement was used to assess the agreement between the methods. We also investigated the experiences of healthcare professionals using digital photographs within the trial.
Digital photographs for 189 out of 240 (79 %) patients in the trial were received for outcome assessment. Of the 189 photographs, 30 (16 %) were uninterpretable. The overall verruca clearance rates were 21 % (43/202,) using the unblinded patient self-reported outcome, 6 % (9/159,) using blinded assessment of digital photographs and 14 % (30/210,) using blinded outcome assessment at the site.
Despite differences in the clearance rates found using different methods of outcome assessment, this did not change the original conclusion of the trial, that there is no evidence of a difference in effectiveness between cryotherapy and salicylic acid. Future trials using digital photographs should consider individual training needs at sites and have a backup method of assessment agreed a priori.
ISRCTN Registry ISRCTN18994246
Blinding is widely used in randomised controlled trials to minimise the possibility of introducing bias as a result of those involved in the study being aware of which treatment has been received . There are several potential issues if patients, investigators and outcome assessors are aware of the treatment allocation. First, patients who know they have received a new treatment may hold either favourable expectations or increased apprehension, whilst those receiving the standard treatment may feel deprived or relieved. Such knowledge may affect psychological or physical responses and could influence the patient’s cooperation in the trial for example attendance for evaluation . Second, investigators who are aware that patients are receiving a novel intervention may follow their progress more closely than those on standard treatment. Alternatively they may transfer their opinion either for or against a treatment to the patient which may affect the patient’s attitude . Finally, for outcome assessors there is the potential to report more favourable outcomes for those in the novel intervention group if they believe it to be a superior treatment. However, this is more likely to be an issue where more subjective outcomes such as pain scores are being assessed rather than objective outcomes such as death.
Trials which have not used appropriate levels of blinding have been shown to report larger effect sizes than blinded studies . It has been suggested that blinded outcome assessment in open trials may in some cases be undertaken by assessment of photographs  and digital photographs have been used previously in randomised controlled trials to assess outcome [6, 7]. Digital photographs are a useful way of capturing and demonstrating global changes in lesions as well as providing objectivity and reproducibility. It may be possible to use computer algorithms to identify the lesion and analyse its size and shape, as well as identify any irregularity of colour and segmentation . There are several advantages to using digital photographs for outcome assessment. First, it eliminates the verbal and non-verbal clues about group allocation and allows the same assessors to evaluate the outcome for all participants in a study. Second, it assists with demonstrating the transparency of the data. Finally, it may assist with centrally monitoring the study, for example confirmation of the existence of patients which may help identify fraudulent behaviour or validate compliance with the trial protocol over the number of treatment visits. Used in conjunction with telecommunications, digital imaging can extend the reach of patient participation in studies. However, a number of disadvantages of using digital photographs also exist, including the purchase cost of equipment, although this has reduced in recent years; the time to take and process photographs; and the need to train staff. Furthermore, additional time is required to handle the photographs at the study coordinating centre.
The EVerT (Effective Verruca Treatments) trial compared the use of cryotherapy delivered by a healthcare professional to self-administered salicylic acid. As such, it was not possible to blind the patients or the healthcare professionals to their treatment allocation. In order to ensure adequate blinding outcome assessments were undertaken in two ways. Firstly a digital photograph of the verruca(e) was taken. As far as we are aware, this is the first study to undertake this type of assessment in such a trial. Second a healthcare professional at the site, who was not involved in treating the participants, assessed the patient to determine whether their verruca(e) was/were still present. It was agreed a priori to use the digital photos as the means of assessment, and assessment at the site would only be used if this was uninterpretable. The aim of this paper is to undertake a secondary analysis to explore the effect of using different methods to assess the primary outcome in the EVerT trial. It also aims to give an overview of the experiences of using digital photographs in the trial in order to inform future dermatology trials.
The EVerT trial
The EVerT trial was a multicentre, two arm randomised controlled open trial evaluating the clinical and cost effectiveness of salicylic acid and cryotherapy for the treatment of verrucae. The study was approved by Trent Multicentre Research Ethics Committee (MREC reference 04/mre04/59), Galway Research Ethics Committee, local research ethics committees, Medicines and Healthcare products Regulatory Agency, Irish Medicines Board and local Research and Development Trusts. All patients provided written informed consent prior to being enrolled in the study. Detailed methods [9, 10] and the main trial results have been published elsewhere . In brief, 240 patients were recruited from University podiatry schools, NHS podiatry clinics and primary care. Patients were eligible for the study if they were over 12 years of age and had at least one verruca which was suitable for treatment with both trial treatments. Patients were randomly allocated to receive cryotherapy using liquid nitrogen or self-treatment with 50 % salicylic acid. The primary outcome was complete clearance of all verrucae at 12 weeks after randomisation as observed on digital photographs by blinded podiatrists and by blinded assessment at the recruiting site by podiatrists, General Practitioners and Practice Nurses. Patient self-reported clearance rates at 12 weeks were also obtained via postal questionnaire.
Digital photographs and outcome assessment methods
Study sites were requested to take digital photographs at baseline and at the 12 week outcome assessment visit. The photographs were taken using either a Nikon Coolpix L11 digital camera provided by the trial, or the Healthcare Professional’s (HCP’s) own camera if it was a similar specification. Sites were given written guidelines on how to use the camera. Digital photographs received by the coordinating centre were reviewed and further advice given by telephone in cases where the quality of the photographs received would make it difficult to interpret. The photographs were assessed by two assessors who were blind to treatment allocation. They independently assessed the photographs for each participant to determine whether the verrucae had cleared. Any discrepancies were referred to a third assessor. The assessors were asked to record if the photograph was uninterpretable and the reasons why and record if they knew for certain which treatment the patient received. Patient self-reported clearance rates at 12 weeks were also obtained via postal questionnaire. Participants who did not attend their 12 week outcome assessment visit were asked to take a digital photograph of their foot and email it to the coordinating centre.
The number of uninterpretable photographs by centre were summarised descriptively. The available outcome assessment data were summarised descriptively along with the baseline data and effectiveness of the blinding.
The verruca clearance rates were reported using the three different methods of assessment and a Cohen’s kappa measure of inter-rater agreement was used to assess the agreement between the methods of assessment. We fitted a logistic regression model with verruca clearance (yes/no) as the primary outcome and treatment, age, type of verruca and previous treatment as covariates. Three analyses were undertaken, one using the digital photograph reported outcome at 12 weeks, one using the blinded outcome assessment at the site at 12 weeks and one using patient’s self-reported outcome at 12 weeks. The results of these analyses were then reported graphically.
All analyses were conducted on an intention-to-treat basis, including all patients in the groups to which they were randomised. All analyses were conducted using Stata version 13.0 (Texas, USA) using two sided significance tests at the 5 % significance level.
Secondary analysis to explore the effect of using different methods of outcome assessment
Number of uninterpretable photographs by centre and type of healthcare professional
Centre ID number
Type of healthcare professional
Number of photographs
Number of uninterpretable photographsa
Take digital photos as part of their routine work
Summary of available outcome assessment data
Number of patients
N = 240
Photograph, blinded assessment at site and patient self-report
Blinded assessment at site and patient self-report (no photograph)
Photograph and blinded assessment at site (no patient self-report)
Photograph and patient self-report (no blinded assessment at site)
Only had photograph (no blinded assessment at site and no patient self-report)
Only had blinded assessment at site (no photograph and no patient self-report)
Only had patient self-report (no photograph and no blinded assessment at site)
Missing data (no photograph, no blinded assessment at site or patient self-report)
Baseline characteristics of participants according to type of outcome assessment
Blinded assessment at site
(N = 159)
(N = 210)
(N = 202)
N, Mean (SD)
158, 30.8 (16.6)
209, 30.7 (16.4)
200, 30.9 (16.7)
Median (min, max)
24.5 (12.0, 75.3)
24.2 (12.0, 75.3)
24.3 (12.0, 75.3)
Type of verrucae
Mosaic, n (%)
Non-mosaic, n (%)
Duration of verrucae (months)
N, Mean (SD)
149, 29.5 (26.7)
199, 26.9 (25.2)
192, 26.9 (25.5)
Median (min, max)
24 (1, 144)
18 (1, 144)
19.1 (1, 144)
Number of verrucae at baseline
N, Mean (SD)
152, 4.0 (5.9)
201, 3.8 (5.5)
194, 3.9 (5.5)
Median (min, max)
2 (1, 55)
2 (1, 55)
2 (1, 55)
Yes, n (%)
No, n (%)
Type of previous treatmenta
Self-treatment, n (%)
Podiatrist/chiropodist, n (%)
GP, n (%)
Trial investigating verruca treatments, n (%)
Other, n (%)
Verruca clearance rates by assessment method
Cryotherapy Number (%)
Salicylic acid Number (%)
Total Number (%)
Blinded assessment at site
The overall verruca clearance rate reported using the patient self-reported outcome data was higher than the rates using both blinded digital photograph assessment and blinded healthcare professional assessment at the site. Although there was little or no evidence of difference in clearance rates using any of the methods of assessment, there was a change in the direction of the effect for the blinded digital photograph assessment compared to the other forms of assessment. Assessment using digital photographs reported slightly higher cure rates with salicylic acid compared to cryotherapy and blinded outcome assessment at the site and patient self-reported outcomes reported higher cure rates with cryotherapy compared to salicylic acid.
Our experiences of using digital photographs
All the patients in this study consented to have photographs taken. Healthcare professionals who either took photographs as part of their routine clinical practice or had an interest in photography had few difficulties with taking and processing the photographs. However, other HCPs found the process challenging, time consuming and often took out of focus photos. This was despite attempts to improve their quality e.g. by reviewing photos at the site whilst the patient was still present and taking multiple photographs. One site which did not routinely take photos as part of their routine work, experienced such difficulties in taking photographs with the camera provided to them by the trial, that they requested assistance from the trust’s medical photographer. Whilst they were willing to take the photographs, this was not practically possible due to their considerable distance from the podiatry department. In 13 cases where patients did not attend their outcome assessment, the coordinating centre wrote to them asking if they would send a digital photograph. One patient complied with this request.
Anecdotally, assessors reviewing the digital photographs reported several reasons why they were unable to interpret them. In some cases the photograph was taken too far away to allow close inspection. However, if the zoom facility was used it sometimes resulted in the image being blurred or distorted. Problems with lighting resulting in glare or shadows also made assessment difficult. Reviewers found it helpful to have more than one concurrent photograph as it either increased the chances of obtaining a better quality photograph or helped in cases where the lesion extended around the foot or where both feet were affected. However, if the baseline photograph was unclear, assessing the follow up photograph for any change was problematic. Assessors found it frustrating not to be able to review the patient in person, as they said it would have helped in their assessment. Although not only an issue for assessment using digital photographs, in some cases it was possible to see that the patient had been treated with salicylic acid due to the circular shaped macerated area surrounding the verrucae, which resulted from treating with the salicylic acid.
Other trials managed by the York Trials Unit have reported additional issues with handling digital photographs. Although it was not an issue in this study, some NHS Trusts only allow medical imaging staff to take photographs of patients. Whilst this improves the quality of the photographs received, it increases costs and the amount of time spent at the clinic for patients. Other NHS Trusts do not allow non-NHS software to be uploaded onto their computers. This resulted in memory cards having to be sent back to the coordinating site.
This study has demonstrated that it is possible to use digital photographs to assess verruca clearance rates. However some potential problems are highlighted that researchers should be aware of. There was a variation between sites as to the quality of the photographs taken and so individualised training needs should be assessed. It also highlights the need to have a back-up option in place in case the image fails. Using a combination of outcome assessment methods allowed us to minimise the amount of missing data in this study. However, the type and priority of data to be used for analysis should be agreed a priori.
As we anticipated the verruca clearance rates were highest when they were self-reported by participants. This could be due partly to the fact that the patients were unblinded to the treatment they received. Some patients allocated to cryotherapy may have reported higher cure rates because they believed the treatment must have been effective as the treatment was painful. Similarly the podiatrists undertaking the blinded outcome assessment both at site and on the photos may have over or under reported clearance rates in those cases where they were able to correctly identify the patient had been treated with salicylic acid. Alternatively it could be due to the healthcare professionals using different criteria on which to define clearance. Healthcare professionals were asked to review whether there was “restoration of normal skin upon close inspection”, whilst patients were asked if their verruca(e) had gone or not and were not given any criteria on which to make this judgement. It may have been possible that patients, who reported their verruca as being unsightly and/or painful, may have used these criteria on which to base whether or not their verruca had been cured, if these signs and symptoms were reduced after treatment. It has been suggested that patients may be able to detect changes in their lesions if they are given the opportunity to review the original lesion on a photograph . The overall cure rate reported by patients may therefore have been lower if this type of assessment had been undertaken. Whilst the clearance rates as assessed by digital photograph or assessment at the site varied considerably this is most likely due to the small numbers involved with the low cure rate. In this case the largest effect size was not seen in the unblinded assessment but in the blinded assessment at the site. Whilst there was a change in the direction of effect when using digital photographs compared to assessment at the site or patient self-reported outcomes, none of the assessment methods resulted in a statistically significant result. The conclusion of the study therefore did not change with the type of assessment used, i.e. there was no evidence of a difference in effectiveness between the treatments. However care should be taken in interpreting these results due to the very low overall cure rates.
Manual tracking of digital photographs by the trial coordinating centre was a time consuming process. However, there are now automated systems, whereby photographs are uploaded directly which would overcome this problem. Whilst it might have been possible to obtain higher quality pictures if using a higher specification of camera there are of course cost and training implications. Whilst technology has improved the potential for taking poor quality photos still remains a possibility. In this study staff often took multiple photos. Whilst these photos were reviewed at the point of taking them, some sites reported that the photograph looked in focus on the LCD screen but when uploaded onto the computer was then found to be out of focus. In future trials, attention needs to be paid to digital image reliability, reproducibility, security and usefulness to ensure issues surrounding authentication, manipulation, audit trail verification and data compression are considered .
In conclusion, although there was a difference in the direction of the effect for verruca clearance rates using data from blinded digital photographs, blinded assessment at the site and unblinded patient self-report, none of the results reached statistical significance. The original conclusion drawn from the study i.e. that there is no evidence of a difference in effectiveness between cryotherapy using liquid nitrogen and daily self-treatment with 50 % salicylic acid is not changed. In future trials we would recommend that if digital photographs are to be used as a means of outcome assessment, then training needs at individual sites should be assessed and more than one photograph should be taken at each time point and an automated process of handling photographs should be used. We would suggest having a backup method of assessment but it should be agreed a priori which data should be used.
EverT, Effective Verruca Treatment; HCP, healthcare professional; MREC, Multicentre Research Ethics Committee.
We thank the patients for taking part in the trial; the podiatrists, GPs and practice nurses for recruiting patients to the study and completing the trial documentation; the principal investigators at each site for coordinating patient recruitment; members of the trial steering committee (Dr Sam Gibbs (independent chair), Dr Jill Mollison, Dr Elaine Thomas and Prof Wesley Vernon); and members of the data monitoring and ethics committee (Dr Anne Maree Keenan (independent chair), Dr Matthew Hankins and Katharine Speaks) for overseeing the study. In addition, we would like to thank Jill Hall, Judith Watson and Farina Hashmi who undertook the blinded outcome assessment of the digital photographs.
This project was funded by the UK National Institute for Health Research Health Technology Assessment Programme (project No 05/513/02). SC, KH, KT, FH and DT all received proportions of their salaries from the NIHR HTA grant in order to conduct the study. CM, FH and MC’s institutions received funding from the NIHR HTA grant to cover the cost of treating trial participants. All other authors declare no support from any organisation for the submitted work. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health. William Ransom & Son Plc supplied the 50 % salicylic acid (Verrugon) at no cost and BOC provided one site with liquid nitrogen storage equipment at reduced cost. These manufacturers had no role in the design of the trial or in the collection, analysis and interpretation of the data.
Availability of data and materials
The data set supporting the results of this article are held at the University of York, York Trials Unit and may be made available upon written request to Mrs Sarah Cockayne email email@example.com.
DT and Jill Hall wrote the original trial protocol. SC, CH, FH, DT and KT were co-applicants on the Health Technology Assessment Programme application and refined the EverT trial protocol. DT and IW were Chief Investigators and oversaw the study. CH designed and undertook the statistical analysis and acts as guarantors for the paper. SC and KH were the trial coordinators and managed the trial on a day to day basis and assisted CH with data validation and data cleaning. CM, FH and MC recruited patients to the study and collected the trial data. JH, JW and JFH undertook the blinded outcome assessment. The initial draft of the manuscript was written by SC but all authors have been involved in revising it critically for important intellectual content. All authors read and approved the final manuscript.
SC, CH, FH, CM, KT and DT were applicants on the NIHR HTA funding application, which included some salary costs. The authors declare that they have no other competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Pocock SJ. Clinical trials: a practical approach. Chichester: John Wiley and Sons; 1983.Google Scholar
- Schulz KF, Chalmers I, Hayes R, Altman D. The landscape and lexicon of blinding in randomized trials. Ann Intern Med. 2002;136:254–59.View ArticlePubMedGoogle Scholar
- Schulz KF, Grimes DA. Blinding in randomised trials: hiding who got what. Lancet. 2002;359:696–700.View ArticlePubMedGoogle Scholar
- Schulz KF, Chalmers I, Hayes R, Altman DR. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408–12.View ArticlePubMedGoogle Scholar
- Day SJ, Altman DG. Statistics notes: blinding in clinical trials and other studies. Br Med J. 2000;321:504.View ArticleGoogle Scholar
- Dumville JC, Worthy G, Soares MO, Bland JM, Cullum N, Dowson C, et al. VenUSII: a randomised controlled trial of larval therapy in the management of leg ulcers. Health Technol Assess. 2009;13:55.View ArticleGoogle Scholar
- Watson JM, Kang’ombe AR, Soares MO, Chuang L-H, Worthy G, Bland JM, et al. VenUS III: a randomised controlled trial of therapeutic ultrasound in the management of venous leg ulcers. Health Technol Assess. 2011;15:13.View ArticleGoogle Scholar
- Jones BF, Plassman P. An instrument to measure the dimensions of skin wounds. IEEE Trans Biomed Eng. 1995;42:5.View ArticleGoogle Scholar
- Cockayne S, Curran M, Denby G, Hashmi F, Hewitt C, Hicks K, et al. EVerT: Cryotherapy versus salicylic acid for the treatment of verrucae: a randomised controlled trial. Health Technol Assess. 2011;15:32.View ArticleGoogle Scholar
- Cockayne S, EVerT Team. The EVERT (effective verruca treatments) trial protocol: a randomised controlled trial to evaluate cryotherapy versus salicylic acid for the treatment of verrucae. Trials. 2010. doi:10.1186/1745-6215-11-12
- Cockayne S, Hewitt C, Hicks K, Jayakody S, Kang’ombe AR, Stamuli E, et al. Cryotherapy versus salicylic acid for the treatment of plantar warts (verrucae): A randomised controlled trial. Br Med J. 2011. doi:10.1136/bmj.d3271
- Oliveria SA, Dusza SW, Phelan DL, Ostroff JS, Berwick M, Halpern AC. Patient adherence to skin self-examination. Effect of nurse intervention with photographs. Am J Prev Med. 2004;26:152–5.View ArticlePubMedGoogle Scholar
- Scheinfeld N. Photographic images, digital imaging, dermatology and the law. Arch Dermatol. 2004. doi: 10.1001/archderm.140.4.473.Google Scholar