Clinical decision support software for diabetic foot risk stratification: development and formative evaluation
© Schoen et al. 2015
Received: 12 May 2015
Accepted: 26 November 2015
Published: 12 December 2015
Identifying people at risk of developing diabetic foot complications is a vital step in prevention programs in primary healthcare settings. Diabetic foot risk stratification systems predict foot ulceration. The aim of this study was to explore the views and experiences of potential end users during development and formative evaluations of an electronic diabetic foot risk stratification tool based on evidence-based guidelines and determine the accuracy of the tool.
Formative evaluation of the risk tool occurred in five stages over an eight-month period and employed a mixed methods research design consisting of semi-structured interviews, focus group and participant observation, online survey, expert review, comparison to the Australian Guidelines and clinical testing.
A total of 43 healthcare practitioners trialled the computerised clinical decision support system during development, with multiple software changes made as a result of feedback. Individual and focus group participants exposed critical design flaws. Live testing revealed risk stratification errors and functional limitations providing the basis for practical improvements. In the final product, all risk calculations and recommendations made by the clinical decision support system reflect current Australian Guidelines.
Development of the computerised clinical decision support system using evidence-based guidelines can be optimised by a multidisciplinary iterative process of feedback, testing and software adaptation by experts in modern development technologies.
KeywordsDiabetes Foot ulcer Diabetic Neuropathies Diabetic Angiopathies Decision Support Systems Clinical
Diabetes is now the leading cause of lower extremity amputation in Australia , with approximately 85 % of lower extremity amputations in people with diabetes preceded by a diabetic foot ulcer [1–5]. The lifetime risk of foot ulceration in people with diabetes is estimated to be between 15 and 25 % [6–8]. Identifying people at risk of foot complications is a crucial step in prevention. Diabetic foot risk stratification predicts foot ulceration and has accordingly become a cornerstone of management .
Diabetic foot risk stratification identifies clinical features of individuals with diabetes that are predictive of the relative risk of foot ulceration in the future. A large number and type of clinical indicators including both systemic and peripheral signs and symptoms have been tested for their predictive value. Systemic features have included age, sex, weight, height, body-mass index, duration of diabetes, type of diabetes, HbA1C, fasting glucose, insulin regimes, history of myocardial infarct, hypertension, erythrocyte sedimentation rate, serum creatinine, kidney disease, eye disease, smoking and alcohol intake . Peripheral features have included peripheral arterial disease, peripheral neuropathy, foot deformity, prior foot ulceration or amputation, abnormal plantar foot pressures, absent tendon reflexes, ankle-brachial index, transcutaneous oxygen tension, lower extremity bypass, intermittent claudication, tinea pedis, onychomycosis, lower leg oedema, dry or fissured skin . Social factors such as level of education, occupation, socioeconomic status, religion, ethnicity and marital status have also been assessed [10–16].
Numerous diabetic foot risk classification systems are described in the literature [7, 9, 17–25]. There is strong evidence to justify risk stratification systems from large cross-sectional and prospective studies [22, 26]. The risk stratification systems have ranged from two to six risk classification groups. Monterio-Soares validated five international risk systems [7, 16, 19, 21, 22] and reported no significant difference between them and all had a high accuracy to detect people who would develop foot ulceration . Most recently the international collaboration, prediction of diabetic foot ulcerations study (PODUS), of more than 16,000 people with diabetes worldwide meta-analysis reported, “the use of a 10-g monofilament or one absent pedal pulse will identify those at moderate or intermediate risk of foot ulceration, and a history of foot ulcers or lower-extremity amputation is sufficient to identify those at high risk” . Notably foot deformity, ethnicity and eye disease were not included in the analysis, as they were not consistently defined in the included data sets.
As a result of expert input and review, in 2011 Australia’s National Health and Medical Research Council (NHMRC) produced National Evidence-Based Guideline on Prevention, Identification and Management of Foot Complications in Diabetes (Guidelines) . This delivered a new national foot risk stratification system, consisting of three levels; low, intermediate and high risk of developing foot complications and the Guideline recommended ‘any trained professional may perform the risk assessment’ and urged the ‘urgent integration of decision support tools into medical software’ . Research has shown that although the procedure for the assessment can be done by any trained professional, the final assessment of level of risk still proved problematic in the absence of additional decision support .
Clinical decision support systems in medicine have progressed from early systems that were never used in a clinical setting to systems that are now integrated into electronic health records across diverse clinical settings [30, 31]. These have been known as artificial intelligence, expert systems or clinical decision support systems (CDSS). CDSS are defined as, “any electronic system designed to aid directly in decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration” . They prompt clinicians through a protocol of pertinent, evidence-based clinical actions and decisions to enhance health-related decisions and actions. CDSS support healthcare professionals to work at a higher level than their standard scope of practice. This is required when there is a workforce shortage of relevant expertise , as occurs in Western Australia where a shortage of podiatrists in rural and remote areas is recognised [34, 35]. Successive systematic reviews have shown the beneficial effect of CDSS on clinical decision making to improve practitioner performance in diagnostic systems, reminder systems, chronic disease management processes of care, drug dosing or prescribing systems, improve rates of screening, and improve adherence to recommended care standards [32, 36–41]. CDSS are increasingly considered to be one of the most effective instruments to improve guideline implementation [38, 42–44]. Trials of CDSS including diabetic foot processes of care have all shown improvements in practitioner performance, rates of screening and adherence to guidelines [45–51]. Most recently, Moja’s systematic review of new generation CDSS in electronic health records reported no reduction in patient mortality but concluded that they might moderately decrease morbidity . Obstacles to wider use of CDSS include poor usability or integration into practitioner workflow, failure to integrate with primary care information technology, the attitudes of end-users, practitioner non-acceptance of computer recommendations, lack of clinician input into the development and their failure to fulfil a perceived clinical need [38, 52, 53].
Diabetic foot risk stratification lends itself to CDSS because it has a strong evidence base, is unambiguous, has explicit input and output criteria for each risk stratification level, each foot risk choice is binary, every possible permutation of foot risks can be decided by an algorithm and produces a correct result. The Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool is a CDSS and has been validated and shown to be predictive of ulceration .
In this study we sought to act on the NHMRC Guideline to integrate CDSS into Australian electronic health records to ensure it fitted within healthcare professionals’ usual workflow . The underpinning question related to whether the use of a CDSS could assist non-podiatrists in risk stratification. The importance of a usable and accurate foot risk assessment CDSS is underpinned by the shortage of podiatrists in rural and remote areas of Western Australia [34, 35].
This paper reports on the process of usability testing that was conducted to develop and evaluate an electronic diabetic foot risk stratification tool for its use in the treatment and management of diabetes in a largely Aboriginal population in Western Australia. The aims of the usability testing were to not only produce a system that was clinically correct, but one that would fit within and enhance the workflow of clinical practice of the health professionals concerned.
Risk tool design
A podiatrist at the Western Australian Centre for Rural Health (DES) and software developers at the Centre for Software Practice at The University of Western Australia (led by DGG) collaborated to develop the risk tool. It was designed to be used by both podiatrists and non-podiatrists for assessing foot risk in people with diabetes in the primary healthcare setting. A usability design process was used incorporating an early focus on users, integrated design, early and continual user testing and iterative design . The term ‘usability’ means getting ‘real’ people to test the system via interviews, observations, surveys. This approach intertwined design and evaluation as users’ feedback was iteratively used to make design changes and improve the system. The risk tool is nested in the cloud-based patient information record system MMEx, which was developed by The Centre for Software Practice in 2007 . MMEx is an e-Health platform providing electronic health records, secure messaging, collaborative care and evidence-based forms with clinical decision support available on mobile platforms. MMEx has been adopted by a broad range of primary, secondary and tertiary public healthcare services, Aboriginal Community Controlled Health Organisations and private practices in several jurisdictions of Australia.
The NHMRC Guidelines were transferred into an online format with CDSS in two steps . Initially, a series of ‘yes/no’ flowcharts of the five clinical inputs was created to make the decisions to decide the risk output. Next, a score to determine the level of risk was assigned with ‘No’ responses scored as zero and ‘yes’ responses scored as one point. If the five risk inputs are all no, then the total score was zero and the risk output was determined as ‘low risk’; if the five risk input score equalled one then the risk output was ‘intermediate risk’; if the risk input score was greater than one, then the risk output was ‘high risk’. Additionally, the presence of a current foot ulcer as a risk input was scored as greater than one so that the output was always high risk. This logic was the basis of the computer code in the development of the tool. The last three elements of the risk tool were practical risk factors that need to be considered in foot documentation although they had not been determined as essential elements of determining the patients’ level of risk. These were the ability to see and reach feet, assessment of footwear and recording of diabetic foot education provided. Their inclusion was to guide clinical practice and prompt health professionals to undertake recommended tasks such as providing written diabetes education or advice on practical risks that need to be considered and addressed.
Phased formative evaluation and iterative development of the risk tool
Formative evaluation of the risk tool occurred in five stages over an 8-month period and employed a mixed methods research design consisting of semi-structured interviews, focus group and participant observation, online survey, expert review, comparison to NHMRC Guidelines and clinical testing . Concurrent qualitative and quantitative data collection procedures were used. Seven healthcare professionals participated in one-to-one semi-structured interviews and observations in Phase One. The second phase was quantitative practical live testing in a routine clinical setting using real patients by the first author (DES) on a tablet in a private rural podiatry practice in Western Australia. The time taken to complete the risk tool within a clinical appointment, language, design and the accuracy of the calculated risk and recommendation were evaluated. Phase Three involved a series of one-to-one semi-structured interviews, observation and one focus group with local experts and potential end users. Twenty local health practitioners (one vascular surgeon, seven podiatrists, four allied health, three general practitioners, three nurses, one nurse practitioner and one Aboriginal Health Worker) participated in one-to-one interviews. They were given web-based access, asked to work with the risk tool, observed during their use and then interviewed. Supplementing this was a focus group discussion with eight diabetes educators. The fourth phase was a quantitative national review of the risk tool by three podiatrists from different states of Australia, who completed an online survey. It contained four scenarios to ascertain if the calculated risk and recommendation were accurate. The participants responded to questions regarding the language, design, and a request for suggestions for improvement of the risk tool. The fifth phase of the evaluation was web-based review by four experts outside of Australia with a publication record in diabetic foot risk stratification systems. They appraised the system for suitability of key criteria and suggestions for improvement.
Ethics approvals for this study were granted by the University of Western Australia, the Western Australian Aboriginal Health Ethics Committee and the Western Australian Country Health Service.
Study population and setting
A total of 43 healthcare professionals participated, 26 novice and 17 experts. Seven in Phase One, one in Phase Two, 28 in Phase Three, three in Phase Four and four in Phase Four. The study was coordinated from Western Australia; Phases One and Two were in rural Western Australia, Phase Three in both rural and urban settings, Phase Four involving participants in three Australian states and Phase Five utilising international reviewers.
Purposive sampling of local healthcare professionals as potential end users of the risk tool and local, national and international experts in diabetic risk stratification systems participated.
Written notes were taken during semi-structured interview and observation of participants’ actions, hesitations or when they needed prompting to complete the risk tool and answers to questions. Live testing risk errors, as judged by the first author based on the NHMRC Guidelines, were recorded in MMEx .
Participants’ comments and observed actions or inaction were thematically categorised into language, design, workflow or risk errors. These were then translated into concrete actions for modification by the MMEx software development team. All suggestions were cross-checked against NHMRC Guideline .
Phase 1: Results of one-to-one interviews and observations
Participants were all supportive of the risk tool concept. Positive comments were that the language was suitable, it was simple to use, took minimal time to find within the electronic patient information system, and it was easy to complete the risk tool. Participants suggested the recommendation for the given level of risk should be displayed at the completion of the risk assessment information being entered.
Phase 2: Live testing
Two hundred and ninety-nine individual risk assessments were completed in a real clinical setting. The risk tool was able to be completed in less than 10 minutes making it suitable for a clinical appointment. The CDSS calculation was fast, taking less than 1 second to calculate the risk and recommendation. Live testing revealed risk errors due to the foot deformity score not initially being calculated and this required further refinement by the MMEx development team. The workflow of the risk tool matched a standard diabetic foot assessment by following the general medical sequencing of a clinical examination process of history, inspection, palpation then specialised procedures with the monofilament. This process also encouraged good infection control procedures to be followed as the history and visual inspection could be completed without gloves on so that interaction with the computer or tablet occurred before donning gloves for the physical assessment. After completing the physical tasks of the risk assessment, hands would be cleaned appropriately and the rest of the tool completed. The self-care and education questions were found to be difficult to deliver without leading the patient in any response/direction and, the need to refine this part of the tool led to initiating the focus group with diabetes educators.
Phase 3: Results of one-to-one interviews and observations
Participant feedback from Phase three
8 Experts and 20 Novices
1. Risk error false negative
1. Forced completion of all sections
c) Subjective terms
d) Incorrect definitions
a) Unchanged, important
a) Risk display at bottom of form
b) Send at bottom of form
c) Boxes around each section
3. Improved Workflow
b) Unchanged/can be incorporated
a) Self-care questions too vague
b) Education questions objective
c) Education questions not specific foot knowledge
a) Simplified; two direct questions
b) Simplified; one direct question
c) One specific foot question
a) Opportunistic education within design
b) Free text section
a) Unchanged/can be incorporated
b) Unchanged/can be incorporated
a) Too many pens in foot deformity
b) Letters in foot deformity
c) Don’t call it deformity
Phase 3: Results of focus group with diabetes educators
Diabetes educators highlighted that the three self-care and three education questions were ambiguous and subjective. They suggested both sets of questions should be changed from the third to the second person point of view and directly ask patients the questions. They suggested that self-care be reduced to two clear questions, directly asking the patient if they could see and reach their feet. The education question was recommended to be simplified to one question only, asking the patient what they understood. This would enable education to be directed to any identified deficits in further foot care education. Additionally, they suggested the clinician should document if verbal or written, or both forms of education were provided. Their final suggestion to improve the tool was to use the features within the form design for opportunistic education, for example, pictures of foot deformities.
Phase 4: Quantitative national review
Positive participant feedback
“…. such a simple, clear, user-friendly tool that is evidence based….”
“Also liked the pulses and neuro + and – buttons, easy to use, but also you’ve made it slow enough so as to limit errors with too many click - well done!”
“I think it’s wonderful and love the concept and design. ….has the potential to be used very widely…..”
“Lovely, enjoyable and useful tool”
Phase 5: International review
International reviewers suggested the current foot risk should be more prominent and include an automatic recall for when the next foot assessment is due. Two experts considered the foot deformity section of the form to be too large and to contain elements (e.g. small muscle wasting) that were unlikely to inform risk assessment. Both inclusions of vibration perception threshold to improve sensitivity and specificity of the tool and, adding eschar to foot deformity and rockerbottom to Charcot foot were suggested. Additionally, they suggested the tool be printable to only one page. Finally, they also suggested using features within the risk tool design for opportunistic education, defining the variables or “Help” section. Positive comments received are included in Table 2.
Diabetic foot risk stratification with CDSS can be integrated into an electronic health record with minimal impact on a podiatrists’ usual workflow. A usability design process with an early focus on end users, integrated design and early and continued user testing was important in recognising major usability issues. Participants’ qualitative responses confirmed the language is suitable, generated extensive feedback for improvement and revealed essential design flaws. Live testing and participants’ quantitative results confirmed the accuracy of the tool. The resultant risk tool is fast, accurate, compatible with the workflow of a diabetic foot assessment and free from false negative errors of risk stratification based on the 2011 NHMRC Guidelines .
The mixed methods approach used in this study has been advocated in the development and evaluation of CDSS [57–63]. This approach was appropriate given the obstacles recognized in the literature that limit the use of CDSS. User-centered design processes can help improve usability and result in more likable computer applications [54, 64]. The naturalistic design of the live testing with real patients for field tests is suggested by Kaplan  and was the approach used to validate the Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool . Measurement of accuracy by peer review and comparison to a standard guideline avoids circularity, and the completeness of results can be supported by triangulation of data from complementary methods comparing data from semi-structured interviews, participant observation, online survey responses, NHMRC Guidelines and clinical testing [57, 60, 62].
Other studies have shown that major usability flaws in CDSS can be recognised by all methods regardless of the expertise of the evaluator as they rely on observation and are easy to perform [53, 66, 67]. Kilsdonk demonstrated a user-centred CDSS design can overcome usability problems when replacing paper-based clinical guidelines into an online format with CDSS as we have done in this study . The Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool, used similar evidence-based guidelines to implement a diabetic foot risk stratification tool in a central web-based database opposed to integrated into an electronic health record as we have done . Our risk tool aligns with Curran’s findings of the clinical reasoning and diagnostic procedures of novice podiatrists of visual cues, touch cues, questions for the patient, and then a diagnostic statement by the podiatrist .
The implications of non-podiatrists using the risk tool have been considered, as CDSS allow healthcare practitioners to work at a higher level of expertise. Studies report that non-podiatrists overreport the presence of foot deformities [29, 69] and are unable to palpate reliably pedal pulses [22, 29, 69]. This would result in a higher risk stratification and is safe for patients with diabetes as the NHMRC recommendation for the higher risk is for referral to a podiatrist for second stage assessment . The recent PODUS study has shown the consistent reliability of the 10-g monofilament regardless of the expertise of the tester, the number of sites and the anatomical sites tested based on five different studies and 11,522 people from three different countries . Finally, the combination of clinical tests for integrated foot risk score as we have used in this study is more sensitive than individual clinical tests for predicting foot ulceration [24, 70, 71].
The strengths of this study were collaboration with an experienced development team, early input from multiple users; the mixed methods approach, and development within a well-established patient information record system . Limitations of this study are the small sample size, no assessment of intraobserver reliability, and that formal usability techniques such as talk aloud protocols or walkthroughs were not used. Furthermore, this study reports the formative evaluation of the risk tool only. Further summative evaluation on the complete system is warranted  and prospective observational studies to determine if the risk tool accurately predicts foot ulceration in a primary healthcare setting.
This study illustrates that the views and experiences of potential end users can be used effectively to develop and evaluate a diabetic foot risk tool with CDSS based on evidence-based guidelines integrated into an electronic health record. The risk tool integrates simple assessment readily available in a clinical setting, and should have minimal impact on experienced healthcare professionals’ usual workflow and reflects current Australian guidelines . It also structures foot examination for those who are less proficient in diabetic foot assessment and will ensure that identified predictors of risk for foot ulcers are referred in a more timely way for definitive assessment and management.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Australian Institute of Health and Welfare. Diabetes: Australian facts 2008. Australian Institute of Health and Welfare, Canberra. 2008. http://www.aihw.gov.au/publications/cvd/daf08/daf08.pdf. Accessed 25 March 2012.
- Pecoraro RE, Reiber GE, Burgess EM. Pathways to diabetic limb amputation. Basis for prevention. Diabetes Care. 1990;13(5):513–21.PubMedView ArticleGoogle Scholar
- Larsson J, Apelqvist J, Agardh CD, Stenstrom A. Decreasing incidence of major amputation in diabetic patients: a consequence of a multidisciplinary foot care team approach? Diabet Med. 1995;12(9):770–6.PubMedView ArticleGoogle Scholar
- Boulton AJ. The diabetic foot: from art to science. The 18th Camillo Golgi lecture. Diabetologia. 2004;47(8):1343–53. doi:https://doi.org/10.1007/s00125-004-1463-y.PubMedView ArticleGoogle Scholar
- National Association of Diabetes Centres. Australian National Diabetes Information Audit & Benchmarking. Sydney. 2011. http://www.health.gov.au/internet/publications/publishing.nsf/Content/pq-diabetes-pubs-andiab11-toc. Accessed February 22, 2014.
- Wild S, Roglic G, Green A, Sicree R, King H. Global prevalence of diabetes: estimates for the year 2000 and projections for 2030. Diabetes Care. 2004;27(5):1047–53.PubMedView ArticleGoogle Scholar
- Boulton AJ, Armstrong DG, Albert SF, Frykberg RG, Hellman R, Kirkman MS, et al. Comprehensive foot examination and risk assessment: a report of the task force of the foot care interest group of the American Diabetes Association, with endorsement by the American Association of Clinical Endocrinologists. Diabetes Care. 2008;31(8):1679–85. doi:https://doi.org/10.2337/dc08-9021.PubMedPubMed CentralView ArticleGoogle Scholar
- Brownrigg JR, Apelqvist J, Bakker K, Schaper NC, Hinchliffe RJ. Evidence-based management of PAD & the diabetic foot. Eur J Vasc Endovasc Surg. 2013;45(6):673–81. doi:https://doi.org/10.1016/j.ejvs.2013.02.014.PubMedView ArticleGoogle Scholar
- Bakker K, Schaper NC, on behalf of the International Working Group on the Diabetic Foot Editorial Board. The development of global consensus guidelines on the management and prevention of the diabetic foot 2011. Diabetes Metab Res Rev. 2012;28(S1):116-8. doi:https://doi.org/10.1002/dmrr.2254.
- Crawford F, Inkster M, Kleijnen J, Fahey T. Predicting foot ulcers in patients with diabetes: a systematic review and meta-analysis. QJM. 2007;100(2):65–86. doi:https://doi.org/10.1093/qjmed/hcl140.PubMedView ArticleGoogle Scholar
- Litzelman DK, Marriott DJ, Vinicor F. Independent physiological predictors of foot lesions in patients with NIDDM. Diabetes Care. 1997;20(8):1273–8.PubMedView ArticleGoogle Scholar
- Bergin SM, Brand CA, Colman PG, Campbell DA. The impact of socio-economic disadvantage on rates of hospital separations for diabetes-related foot disease in Victoria, Australia. J Foot Ankle Res. 2011;4:17. doi:https://doi.org/10.1186/1757-1146-4-17.PubMedPubMed CentralView ArticleGoogle Scholar
- Leese GP, Feng Z, Leese RM, Dibben C, Emslie-Smith A. Impact of health-care accessibility and social deprivation on diabetes related foot disease. Diabet Med. 2013;30(4):484–90. doi:https://doi.org/10.1111/dme.12108.PubMedView ArticleGoogle Scholar
- McNeely MJ, Boyko EJ, Ahroni JH, Stensel VL, Reiber GE, Smith DG, et al. The independent contributions of diabetic neuropathy and vasculopathy in foot ulceration. How great are the risks? Diabetes Care. 1995;18(2):216–9.PubMedView ArticleGoogle Scholar
- Sriussadaporn S, Mekanandha P, Vannasaeng S, Nitiyanant W, Komoltri C, Ploybutr S, et al. Factors associated with diabetic foot ulceration in Thailand: a case-control study. Diabet Med. 1997;14(1):50–6. doi:https://doi.org/10.1002/(SICI)1096-9136(199701)14%3A1%3C50%3A%3AAID-DIA292%3E3.0.CO%3B2-6.PubMedView ArticleGoogle Scholar
- Boyko EJ, Ahroni JH, Cohen V, Nelson KM, Heagerty PJ. Prediction of diabetic foot ulcer occurrence using commonly available clinical information: the Seattle Diabetic Foot Study. Diabetes Care. 2006;29(6):1202–7. doi:https://doi.org/10.2337/dc05-2031.PubMedView ArticleGoogle Scholar
- Rith-Najarian SJ, Stolusky T, Gohdes DM. Identifying diabetic patients at high risk for lower-extremity amputation in a primary health care setting. A prospective evaluation of simple screening criteria. Diabetes Care. 1992;15(10):1386–9.PubMedView ArticleGoogle Scholar
- Mayfield JA, Reiber GE, Nelson RG, Greene T. A foot risk classification system to predict diabetic amputation in Pima Indians. Diabetes Care. 1996;19(7):704–9.PubMedView ArticleGoogle Scholar
- Lavery LA, Armstrong DG, Vela SA, Quebedeaux TL, Fleischli JG. Practical criteria for screening patients at high risk for diabetic foot ulceration. Arch Intern Med. 1998;158(2):157–62. doi:https://doi.org/10.1001/archinte.158.2.157.PubMedView ArticleGoogle Scholar
- Peters EJ, Lavery LA, International Working Group on the Diabetic F. Effectiveness of the diabetic foot risk classification system of the International Working Group on the Diabetic Foot. Diabetes Care. 2001;24(8):1442–7.PubMedView ArticleGoogle Scholar
- Lavery LA, Peters EJ, Williams JR, Murdoch DP, Hudson A, Lavery DC, et al. Reevaluating the way we classify the diabetic foot: restructuring the diabetic foot risk classification system of the International Working Group on the Diabetic Foot. Diabetes Care. 2008;31(1):154–6. doi:https://doi.org/10.2337/dc07-1302.PubMedView ArticleGoogle Scholar
- Leese GP, Reid F, Green V, McAlpine R, Cunningham S, Emslie-Smith AM, et al. Stratification of foot ulcer risk in patients with diabetes: a population-based study. Int J Clin Pract. 2006;60(5):541–5. doi:https://doi.org/10.1111/j.1368-5031.2006.00899.x.PubMedView ArticleGoogle Scholar
- National Institute for Health and Clinical Excellence. Diabetic foot problems: prevention and management National Institute for Health and Clinical Excellence, London. 2015. http://www.nice.org.uk/guidance/ng19. Accessed 22 Sept 2015.
- National Health and Medical Research Council. National Evidence-Based Guideline: Prevention, Identification and Management of Foot Complications in Diabetes. (Part of the Guidelines on the management of Type 2 Diabetes). National Health and Medical Research Council, Melbourne. 2011. http://t2dgr.bakeridi.edu.au/LinkClick.aspx?fileticket=Z_vePysE8Ng%3D&tabid=760&mid=1629. Accessed 10 Apr 2011.
- Scottish Intercollegiate Guidelines Network. Management of diabetes A national clinical guideline. Scottish Intercollegiate Guidelines Network, Edinburgh. 2010. http://www.sign.ac.uk/pdf/sign116.pdf. Accessed 5 Apr 2011.
- Abbott CA, Carrington AL, Ashe H, Bath S, Every LC, Griffiths J, et al. The North-West Diabetes Foot Care Study: incidence of, and risk factors for, new diabetic foot ulceration in a community-based patient cohort. Diabet Med. 2002;19(5):377–84.PubMedView ArticleGoogle Scholar
- Monteiro-Soares M, Vaz-Carneiro A, Sampaio S, Dinis-Ribeiro M. Validation and comparison of currently available stratification systems for patients with diabetes by risk of foot ulcer development. Eur J Endocrinol. 2012;167(3):401–7. doi:https://doi.org/10.1530/EJE-12-0279.PubMedView ArticleGoogle Scholar
- Crawford F, Cezard G, Chappell FM, Murray GD, Price JF, Sheikh A, et al. A systematic review and individual patient data meta-analysis of prognostic factors for foot ulceration in people with diabetes: the international research collaboration for the prediction of diabetic foot ulcerations (PODUS). Health Technol Assess. 2015;19(57):1–210. doi:https://doi.org/10.3310/hta19570.PubMedPubMed CentralView ArticleGoogle Scholar
- Bower VM, Hobbs M. Validation of the basic foot screening checklist: a population screening tool for identifying foot ulcer risk in people with diabetes mellitus. J Am Podiatr Med Assoc. 2009;99(4):339–47.PubMedView ArticleGoogle Scholar
- Shortliffe EH, Davis R, Axline SG, Buchanan BG, Green CC, Cohen SN. Computer-based consultations in clinical therapeutics: explanation and rule acquisition capabilities of the MYCIN system. Comput Biomed Res. 1975;8(4):303–20.PubMedView ArticleGoogle Scholar
- Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health. 2014;104(12):e12–22. doi:https://doi.org/10.2105/AJPH.2014.302164.PubMedPubMed CentralView ArticleGoogle Scholar
- Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi:https://doi.org/10.1136/bmj.38398.500764.8F.PubMedPubMed CentralView ArticleGoogle Scholar
- Durkin J. Expert systems:Design and Development. London: Englewood Cliffs; 1994.Google Scholar
- Department of Health Western Australia. High Risk Foot Model of Care. Health Networks Branch, Perth. 2010. http://www.healthnetworks.health.wa.gov.au/modelsofcare/docs/High_Risk_Foot_Model_of_Care.pdf. Accessed 17 Jun 2010.
- Department of Health Western Australia. Diabetes in Western Australia: Prevalence and services in 2012. Health Networks Branch, Perth. 2012. http://www.healthnetworks.health.wa.gov.au/docs/WA_diabetes_services_review_Report.pdf. Accessed 16 Apr 2013.
- Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120(2):135–42.PubMedView ArticleGoogle Scholar
- Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339–46.PubMedView ArticleGoogle Scholar
- Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38. doi:https://doi.org/10.1001/jama.293.10.1223.PubMedView ArticleGoogle Scholar
- Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29–43. doi:https://doi.org/10.7326/0003-4819-157-1-201207030-00450.PubMedView ArticleGoogle Scholar
- Roshanov PS, Misra S, Gerstein HC, Garg AX, Sebaldt RJ, Mackay JA, et al. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011;6:92. doi:https://doi.org/10.1186/1748-5908-6-92.PubMedPubMed CentralView ArticleGoogle Scholar
- Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. doi:https://doi.org/10.1136/bmj.f657.PubMedView ArticleGoogle Scholar
- Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):iii–iv. 1-72.PubMedView ArticleGoogle Scholar
- Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362(9391):1225–30. doi:https://doi.org/10.1016/S0140-6736(03)14546-1.PubMedView ArticleGoogle Scholar
- Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999;6(2):104–14.PubMedPubMed CentralView ArticleGoogle Scholar
- Nilasena DS, Lincoln MJ. A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines. Proc Annu Symp Comput Appl Med Care. 1995;640:5.Google Scholar
- Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. Am J Med. 1997;102(1):89–98.PubMedView ArticleGoogle Scholar
- Montori VM, Dinneen SF, Gorman CA, Zimmerman BR, Rizza RA, Bjornsen SS, et al. The impact of planned care and a diabetes electronic management system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. Diabetes Care. 2002;25(11):1952–7.PubMedView ArticleGoogle Scholar
- Meigs JB, Cagliero E, Dubey A, Murphy-Sheehy P, Gildesgame C, Chueh H, et al. A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project. Diabetes Care. 2003;26(3):750–7.PubMedView ArticleGoogle Scholar
- Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, et al. Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA. 2000;284(11):1411–6.PubMedView ArticleGoogle Scholar
- Khoury A, Landers P, Roth M, Rowe N, DaMert G, Dahar W, et al. Computer-supported identification and intervention for diabetic patients at risk for amputation. MD Computing. 1998;15(5):307–10.PubMedGoogle Scholar
- Peterson KA, Radosevich DM, O’Connor PJ, Nyman JA, Prineas RJ, Smith SA, et al. Improving Diabetes Care in Practice: findings from the TRANSLATE trial. Diabetes Care. 2008;31(12):2238–43. doi:https://doi.org/10.2337/dc08-2034.PubMedPubMed CentralView ArticleGoogle Scholar
- Crawford F, Bekker HL, Young M, Sheikh A. General practitioners’ and nurses’ experiences of using computerised decision support in screening for diabetic foot disease: implementing Scottish Clinical Information - Diabetes Care in routine clinical practice. Inform Prim Care. 2010;18(4):259–68.PubMedGoogle Scholar
- Carroll C, Marsden P, Soden P, Naylor E, New J, Dornan T. Involving users in the design and usability evaluation of a clinical decision support system. Comput Methods Programs Biomed. 2002;69(2):123–35.PubMedView ArticleGoogle Scholar
- Gould JD, Boies SJ, Lewis C. Making Usable, Useful, Productivity - Enhancing Computer-Applications. Commun ACM. 1991;34(1):74–85. doi:https://doi.org/10.1145/99977.99993.View ArticleGoogle Scholar
- MMEx eHealth Platform. Perth. 2007. http://www.mmex.net.au/Products.aspx. Accessed 11 March, 2011.
- Tashakkori A, Teddlie C. Handbook of mixed methods in social & behavioral research. Thousand Oaks: SAGE Publications; 2003.Google Scholar
- Ammenwerth E, Iller C, Mansmann U. Can evaluation studies benefit from triangulation? A case study. Int J Med Inform. 2003;70(2-3):237–48. doi:https://doi.org/10.1016/s1386-5056(03)00059-5.PubMedView ArticleGoogle Scholar
- Kaplan B. Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform. 2001;64(1):39–56. doi:https://doi.org/10.1016/S1386-5056(01)00184-8.PubMedView ArticleGoogle Scholar
- Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med. 2009;6(8):e1000126. doi:https://doi.org/10.1371/journal.pmed.1000126.PubMedPubMed CentralView ArticleGoogle Scholar
- Wyatt J. Quantitative evaluation of clinical software, exemplified by decision support systems. Int J Med Inform. 1997;47(3):165–73.PubMedView ArticleGoogle Scholar
- Kaplan B, Duchon D. Combining Qualitative and Quantitative Methods in Information-Systems Research - a Case-Study. MIS Q. 1988;12(4):571–86. doi:https://doi.org/10.2307/249133.View ArticleGoogle Scholar
- Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med Inform (Lond). 1990;15(3):205–17.View ArticleGoogle Scholar
- Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8(6):527–34.PubMedPubMed CentralView ArticleGoogle Scholar
- Kilsdonk E, Peute LW, Riezebos RJ, Kremer LC, Jaspers MW. From an expert-driven paper guideline to a user-centred decision support system: a usability comparison study. Artif Intell Med. 2013;59(1):5–13. doi:https://doi.org/10.1016/j.artmed.2013.04.004.PubMedView ArticleGoogle Scholar
- Kaplan B. Evaluating informatics applications--clinical decision support systems literature review. Int J Med Inform. 2001;64(1):15–37. doi:https://doi.org/10.1016/S1386-5056(01)00183-6.PubMedView ArticleGoogle Scholar
- Nielsen J. Applying Discount Usability Engineering. Ieee Softw. 1995;12(1):98–100. doi:https://doi.org/10.1109/52.363161.View ArticleGoogle Scholar
- Leslie SJ, Hartswood M, Meurig C, McKee SP, Slack R, Procter R, et al. Clinical decision support software for management of chronic heart failure: development and evaluation. Comput Biol Med. 2006;36(5):495–506. doi:https://doi.org/10.1016/j.compbiomed.2005.02.002.PubMedView ArticleGoogle Scholar
- Curran MJ, Campbell J, Rugg G. An investigation into the clinical reasoning of both expert and novice podiatrists. Foot. 2006;16(1):28–32. doi:https://doi.org/10.1016/j.foot.2005.11.001.View ArticleGoogle Scholar
- Edelman D, Sanders LJ, Pogach L. Reproducibility and accuracy among primary care providers of a screening examination for foot ulcer risk among diabetic patients. Prev Med. 1998;27(2):274–8. doi:https://doi.org/10.1006/pmed.1998.0263.PubMedView ArticleGoogle Scholar
- Leese GP, Cochrane L, Mackie AD, Stang D, Brown K, Green V. Measuring the accuracy of different ways to identify the ‘at-risk’ foot in routine clinical practice. Diabet Med. 2011;28(6):747–54. doi:https://doi.org/10.1111/j.1464-5491.2011.03297.x.PubMedView ArticleGoogle Scholar
- National Health and Medical Research Council. Prevention, Identification and Management of Foot Complications in Diabetes Technical Report. National Health and Medical Research Council, Melbourne 2011. http://t2dgr.bakeridi.edu.au/LinkClick.aspx?fileticket=IGRCbi9Seqs%3d&tabid=172. Accessed 10 Jun 2011.