Skip to main content
Download PDF

Clinical decision support software for diabetic foot risk stratification: development and formative evaluation

Journal of Foot and Ankle Research volume 8, Article number: 73 (2015) Cite this article

Abstract

Background

Identifying people at risk of developing diabetic foot complications is a vital step in prevention programs in primary healthcare settings. Diabetic foot risk stratification systems predict foot ulceration. The aim of this study was to explore the views and experiences of potential end users during development and formative evaluations of an electronic diabetic foot risk stratification tool based on evidence-based guidelines and determine the accuracy of the tool.

Methods

Formative evaluation of the risk tool occurred in five stages over an eight-month period and employed a mixed methods research design consisting of semi-structured interviews, focus group and participant observation, online survey, expert review, comparison to the Australian Guidelines and clinical testing.

Results

A total of 43 healthcare practitioners trialled the computerised clinical decision support system during development, with multiple software changes made as a result of feedback. Individual and focus group participants exposed critical design flaws. Live testing revealed risk stratification errors and functional limitations providing the basis for practical improvements. In the final product, all risk calculations and recommendations made by the clinical decision support system reflect current Australian Guidelines.

Conclusions

Development of the computerised clinical decision support system using evidence-based guidelines can be optimised by a multidisciplinary iterative process of feedback, testing and software adaptation by experts in modern development technologies.

Peer Review reports

Background

Diabetes is now the leading cause of lower extremity amputation in Australia [1], with approximately 85 % of lower extremity amputations in people with diabetes preceded by a diabetic foot ulcer [1–5]. The lifetime risk of foot ulceration in people with diabetes is estimated to be between 15 and 25 % [6–8]. Identifying people at risk of foot complications is a crucial step in prevention. Diabetic foot risk stratification predicts foot ulceration and has accordingly become a cornerstone of management [9].

Diabetic foot risk stratification identifies clinical features of individuals with diabetes that are predictive of the relative risk of foot ulceration in the future. A large number and type of clinical indicators including both systemic and peripheral signs and symptoms have been tested for their predictive value. Systemic features have included age, sex, weight, height, body-mass index, duration of diabetes, type of diabetes, HbA1C, fasting glucose, insulin regimes, history of myocardial infarct, hypertension, erythrocyte sedimentation rate, serum creatinine, kidney disease, eye disease, smoking and alcohol intake [10]. Peripheral features have included peripheral arterial disease, peripheral neuropathy, foot deformity, prior foot ulceration or amputation, abnormal plantar foot pressures, absent tendon reflexes, ankle-brachial index, transcutaneous oxygen tension, lower extremity bypass, intermittent claudication, tinea pedis, onychomycosis, lower leg oedema, dry or fissured skin [10]. Social factors such as level of education, occupation, socioeconomic status, religion, ethnicity and marital status have also been assessed [10–16].

Numerous diabetic foot risk classification systems are described in the literature [7, 9, 17–25]. There is strong evidence to justify risk stratification systems from large cross-sectional and prospective studies [22, 26]. The risk stratification systems have ranged from two to six risk classification groups. Monterio-Soares validated five international risk systems [7, 16, 19, 21, 22] and reported no significant difference between them and all had a high accuracy to detect people who would develop foot ulceration [27]. Most recently the international collaboration, prediction of diabetic foot ulcerations study (PODUS), of more than 16,000 people with diabetes worldwide meta-analysis reported, “the use of a 10-g monofilament or one absent pedal pulse will identify those at moderate or intermediate risk of foot ulceration, and a history of foot ulcers or lower-extremity amputation is sufficient to identify those at high risk” [28]. Notably foot deformity, ethnicity and eye disease were not included in the analysis, as they were not consistently defined in the included data sets.

As a result of expert input and review, in 2011 Australia’s National Health and Medical Research Council (NHMRC) produced National Evidence-Based Guideline on Prevention, Identification and Management of Foot Complications in Diabetes (Guidelines) [24]. This delivered a new national foot risk stratification system, consisting of three levels; low, intermediate and high risk of developing foot complications and the Guideline recommended ‘any trained professional may perform the risk assessment’ and urged the ‘urgent integration of decision support tools into medical software’ [24]. Research has shown that although the procedure for the assessment can be done by any trained professional, the final assessment of level of risk still proved problematic in the absence of additional decision support [29].

Clinical decision support systems in medicine have progressed from early systems that were never used in a clinical setting to systems that are now integrated into electronic health records across diverse clinical settings [30, 31]. These have been known as artificial intelligence, expert systems or clinical decision support systems (CDSS). CDSS are defined as, “any electronic system designed to aid directly in decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration” [32]. They prompt clinicians through a protocol of pertinent, evidence-based clinical actions and decisions to enhance health-related decisions and actions. CDSS support healthcare professionals to work at a higher level than their standard scope of practice. This is required when there is a workforce shortage of relevant expertise [33], as occurs in Western Australia where a shortage of podiatrists in rural and remote areas is recognised [34, 35]. Successive systematic reviews have shown the beneficial effect of CDSS on clinical decision making to improve practitioner performance in diagnostic systems, reminder systems, chronic disease management processes of care, drug dosing or prescribing systems, improve rates of screening, and improve adherence to recommended care standards [32, 36–41]. CDSS are increasingly considered to be one of the most effective instruments to improve guideline implementation [38, 42–44]. Trials of CDSS including diabetic foot processes of care have all shown improvements in practitioner performance, rates of screening and adherence to guidelines [45–51]. Most recently, Moja’s systematic review of new generation CDSS in electronic health records reported no reduction in patient mortality but concluded that they might moderately decrease morbidity [31]. Obstacles to wider use of CDSS include poor usability or integration into practitioner workflow, failure to integrate with primary care information technology, the attitudes of end-users, practitioner non-acceptance of computer recommendations, lack of clinician input into the development and their failure to fulfil a perceived clinical need [38, 52, 53].

Diabetic foot risk stratification lends itself to CDSS because it has a strong evidence base, is unambiguous, has explicit input and output criteria for each risk stratification level, each foot risk choice is binary, every possible permutation of foot risks can be decided by an algorithm and produces a correct result. The Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool is a CDSS and has been validated and shown to be predictive of ulceration [22].

In this study we sought to act on the NHMRC Guideline to integrate CDSS into Australian electronic health records to ensure it fitted within healthcare professionals’ usual workflow [24]. The underpinning question related to whether the use of a CDSS could assist non-podiatrists in risk stratification. The importance of a usable and accurate foot risk assessment CDSS is underpinned by the shortage of podiatrists in rural and remote areas of Western Australia [34, 35].

This paper reports on the process of usability testing that was conducted to develop and evaluate an electronic diabetic foot risk stratification tool for its use in the treatment and management of diabetes in a largely Aboriginal population in Western Australia. The aims of the usability testing were to not only produce a system that was clinically correct, but one that would fit within and enhance the workflow of clinical practice of the health professionals concerned.

Method

Risk tool design

A podiatrist at the Western Australian Centre for Rural Health (DES) and software developers at the Centre for Software Practice at The University of Western Australia (led by DGG) collaborated to develop the risk tool. It was designed to be used by both podiatrists and non-podiatrists for assessing foot risk in people with diabetes in the primary healthcare setting. A usability design process was used incorporating an early focus on users, integrated design, early and continual user testing and iterative design [54]. The term ‘usability’ means getting ‘real’ people to test the system via interviews, observations, surveys. This approach intertwined design and evaluation as users’ feedback was iteratively used to make design changes and improve the system. The risk tool is nested in the cloud-based patient information record system MMEx, which was developed by The Centre for Software Practice in 2007 [55]. MMEx is an e-Health platform providing electronic health records, secure messaging, collaborative care and evidence-based forms with clinical decision support available on mobile platforms. MMEx has been adopted by a broad range of primary, secondary and tertiary public healthcare services, Aboriginal Community Controlled Health Organisations and private practices in several jurisdictions of Australia.

The risk tool is based on the NHMRC Guidelines clinical inputs of (1) previous amputation, (2) previous foot ulcer, (3) foot deformity, (4) pulses, and (5) peripheral neuropathy [24]. The CDSS risk output appears as the words low, intermediate or high risk and is accompanied by the NHMRC recommendation for each given level of risk [24]. For example, intermediate risk suggests “Review 6 months and refer for foot protection program (includes foot care education, podiatry review and appropriate footwear” (Fig. 1).

Fig. 1
figure 1

The risk tool

Full size image

The NHMRC Guidelines were transferred into an online format with CDSS in two steps [24]. Initially, a series of ‘yes/no’ flowcharts of the five clinical inputs was created to make the decisions to decide the risk output. Next, a score to determine the level of risk was assigned with ‘No’ responses scored as zero and ‘yes’ responses scored as one point. If the five risk inputs are all no, then the total score was zero and the risk output was determined as ‘low risk’; if the five risk input score equalled one then the risk output was ‘intermediate risk’; if the risk input score was greater than one, then the risk output was ‘high risk’. Additionally, the presence of a current foot ulcer as a risk input was scored as greater than one so that the output was always high risk. This logic was the basis of the computer code in the development of the tool. The last three elements of the risk tool were practical risk factors that need to be considered in foot documentation although they had not been determined as essential elements of determining the patients’ level of risk. These were the ability to see and reach feet, assessment of footwear and recording of diabetic foot education provided. Their inclusion was to guide clinical practice and prompt health professionals to undertake recommended tasks such as providing written diabetes education or advice on practical risks that need to be considered and addressed.

Phased formative evaluation and iterative development of the risk tool

Study design

Formative evaluation of the risk tool occurred in five stages over an 8-month period and employed a mixed methods research design consisting of semi-structured interviews, focus group and participant observation, online survey, expert review, comparison to NHMRC Guidelines and clinical testing [56]. Concurrent qualitative and quantitative data collection procedures were used. Seven healthcare professionals participated in one-to-one semi-structured interviews and observations in Phase One. The second phase was quantitative practical live testing in a routine clinical setting using real patients by the first author (DES) on a tablet in a private rural podiatry practice in Western Australia. The time taken to complete the risk tool within a clinical appointment, language, design and the accuracy of the calculated risk and recommendation were evaluated. Phase Three involved a series of one-to-one semi-structured interviews, observation and one focus group with local experts and potential end users. Twenty local health practitioners (one vascular surgeon, seven podiatrists, four allied health, three general practitioners, three nurses, one nurse practitioner and one Aboriginal Health Worker) participated in one-to-one interviews. They were given web-based access, asked to work with the risk tool, observed during their use and then interviewed. Supplementing this was a focus group discussion with eight diabetes educators. The fourth phase was a quantitative national review of the risk tool by three podiatrists from different states of Australia, who completed an online survey. It contained four scenarios to ascertain if the calculated risk and recommendation were accurate. The participants responded to questions regarding the language, design, and a request for suggestions for improvement of the risk tool. The fifth phase of the evaluation was web-based review by four experts outside of Australia with a publication record in diabetic foot risk stratification systems. They appraised the system for suitability of key criteria and suggestions for improvement.

Ethics approvals for this study were granted by the University of Western Australia, the Western Australian Aboriginal Health Ethics Committee and the Western Australian Country Health Service.

Study population and setting

A total of 43 healthcare professionals participated, 26 novice and 17 experts. Seven in Phase One, one in Phase Two, 28 in Phase Three, three in Phase Four and four in Phase Four. The study was coordinated from Western Australia; Phases One and Two were in rural Western Australia, Phase Three in both rural and urban settings, Phase Four involving participants in three Australian states and Phase Five utilising international reviewers.

Sampling

Purposive sampling of local healthcare professionals as potential end users of the risk tool and local, national and international experts in diabetic risk stratification systems participated.

Data collection

Written notes were taken during semi-structured interview and observation of participants’ actions, hesitations or when they needed prompting to complete the risk tool and answers to questions. Live testing risk errors, as judged by the first author based on the NHMRC Guidelines, were recorded in MMEx [55].

Data analysis

Participants’ comments and observed actions or inaction were thematically categorised into language, design, workflow or risk errors. These were then translated into concrete actions for modification by the MMEx software development team. All suggestions were cross-checked against NHMRC Guideline [24].

Results

Phase 1: Results of one-to-one interviews and observations

Participants were all supportive of the risk tool concept. Positive comments were that the language was suitable, it was simple to use, took minimal time to find within the electronic patient information system, and it was easy to complete the risk tool. Participants suggested the recommendation for the given level of risk should be displayed at the completion of the risk assessment information being entered.

Phase 2: Live testing

Two hundred and ninety-nine individual risk assessments were completed in a real clinical setting. The risk tool was able to be completed in less than 10 minutes making it suitable for a clinical appointment. The CDSS calculation was fast, taking less than 1 second to calculate the risk and recommendation. Live testing revealed risk errors due to the foot deformity score not initially being calculated and this required further refinement by the MMEx development team. The workflow of the risk tool matched a standard diabetic foot assessment by following the general medical sequencing of a clinical examination process of history, inspection, palpation then specialised procedures with the monofilament. This process also encouraged good infection control procedures to be followed as the history and visual inspection could be completed without gloves on so that interaction with the computer or tablet occurred before donning gloves for the physical assessment. After completing the physical tasks of the risk assessment, hands would be cleaned appropriately and the rest of the tool completed. The self-care and education questions were found to be difficult to deliver without leading the patient in any response/direction and, the need to refine this part of the tool led to initiating the focus group with diabetes educators.

Phase 3: Results of one-to-one interviews and observations

Five language concerns were raised. Three concerns were easily solved by correcting spelling, removing subjective qualifiers (e.g. very/superficial) and incorrect definitions. The other two issues were terminology. Participants felt “Send as HL7” was meaningless to the user, and queried its necessity. Secondly, they suggested the “high risk” recommendation should be “Refer to a High Risk Foot Podiatrist.” Participants also suggested a free text section was needed. A major workflow issue was exposed by observation of participants. They were unable to complete the risk tool as the “Calculate Risk” button was at the top of the risk tool only and resulted in participants thinking the CDSS had not worked. A critical design flaw of the CDSS was revealed. A false negative error occurred if the entire risk tool was not completed. A false negative is an error in which a test result improperly indicates no presence of a condition (the result is negative) when in reality it is present. This occurred during testing if one section, for example, pulses, was not completed and results in an incorrect risk stratification (Table 1).

Table 1 Participant feedback from Phase three
Full size table

Phase 3: Results of focus group with diabetes educators

Diabetes educators highlighted that the three self-care and three education questions were ambiguous and subjective. They suggested both sets of questions should be changed from the third to the second person point of view and directly ask patients the questions. They suggested that self-care be reduced to two clear questions, directly asking the patient if they could see and reach their feet. The education question was recommended to be simplified to one question only, asking the patient what they understood. This would enable education to be directed to any identified deficits in further foot care education. Additionally, they suggested the clinician should document if verbal or written, or both forms of education were provided. Their final suggestion to improve the tool was to use the features within the form design for opportunistic education, for example, pictures of foot deformities.

Phase 4: Quantitative national review

National experts confirmed the accuracy of the risk stratification output. However, the recommendations were incomplete and not the exact wording of the NHMRC Guideline [24]. The language of the risk and recommendation for a current ulcer was problematic. They suggested the risk and recommendation be referenced and a web link to the NHMRC Guidelines be included [24]. National experts suggested the workflow could be altered, to have the assessment of pulses and neuropathy sections first, given their greater importance. They also suggested having a free text section. Positive comments are shown in Table 2.

Table 2 Positive participant feedback
Full size table

Phase 5: International review

International reviewers suggested the current foot risk should be more prominent and include an automatic recall for when the next foot assessment is due. Two experts considered the foot deformity section of the form to be too large and to contain elements (e.g. small muscle wasting) that were unlikely to inform risk assessment. Both inclusions of vibration perception threshold to improve sensitivity and specificity of the tool and, adding eschar to foot deformity and rockerbottom to Charcot foot were suggested. Additionally, they suggested the tool be printable to only one page. Finally, they also suggested using features within the risk tool design for opportunistic education, defining the variables or “Help” section. Positive comments received are included in Table 2.

Discussion

Diabetic foot risk stratification with CDSS can be integrated into an electronic health record with minimal impact on a podiatrists’ usual workflow. A usability design process with an early focus on end users, integrated design and early and continued user testing was important in recognising major usability issues. Participants’ qualitative responses confirmed the language is suitable, generated extensive feedback for improvement and revealed essential design flaws. Live testing and participants’ quantitative results confirmed the accuracy of the tool. The resultant risk tool is fast, accurate, compatible with the workflow of a diabetic foot assessment and free from false negative errors of risk stratification based on the 2011 NHMRC Guidelines [24].

The mixed methods approach used in this study has been advocated in the development and evaluation of CDSS [57–63]. This approach was appropriate given the obstacles recognized in the literature that limit the use of CDSS. User-centered design processes can help improve usability and result in more likable computer applications [54, 64]. The naturalistic design of the live testing with real patients for field tests is suggested by Kaplan [65] and was the approach used to validate the Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool [22]. Measurement of accuracy by peer review and comparison to a standard guideline avoids circularity, and the completeness of results can be supported by triangulation of data from complementary methods comparing data from semi-structured interviews, participant observation, online survey responses, NHMRC Guidelines and clinical testing [57, 60, 62].

Other studies have shown that major usability flaws in CDSS can be recognised by all methods regardless of the expertise of the evaluator as they rely on observation and are easy to perform [53, 66, 67]. Kilsdonk demonstrated a user-centred CDSS design can overcome usability problems when replacing paper-based clinical guidelines into an online format with CDSS as we have done in this study [64]. The Scottish Care Information – Diabetes Collaboration system’s diabetic foot electronic decision support tool, used similar evidence-based guidelines to implement a diabetic foot risk stratification tool in a central web-based database opposed to integrated into an electronic health record as we have done [22]. Our risk tool aligns with Curran’s findings of the clinical reasoning and diagnostic procedures of novice podiatrists of visual cues, touch cues, questions for the patient, and then a diagnostic statement by the podiatrist [68].

The implications of non-podiatrists using the risk tool have been considered, as CDSS allow healthcare practitioners to work at a higher level of expertise. Studies report that non-podiatrists overreport the presence of foot deformities [29, 69] and are unable to palpate reliably pedal pulses [22, 29, 69]. This would result in a higher risk stratification and is safe for patients with diabetes as the NHMRC recommendation for the higher risk is for referral to a podiatrist for second stage assessment [24]. The recent PODUS study has shown the consistent reliability of the 10-g monofilament regardless of the expertise of the tester, the number of sites and the anatomical sites tested based on five different studies and 11,522 people from three different countries [28]. Finally, the combination of clinical tests for integrated foot risk score as we have used in this study is more sensitive than individual clinical tests for predicting foot ulceration [24, 70, 71].

The strengths of this study were collaboration with an experienced development team, early input from multiple users; the mixed methods approach, and development within a well-established patient information record system [55]. Limitations of this study are the small sample size, no assessment of intraobserver reliability, and that formal usability techniques such as talk aloud protocols or walkthroughs were not used. Furthermore, this study reports the formative evaluation of the risk tool only. Further summative evaluation on the complete system is warranted [62] and prospective observational studies to determine if the risk tool accurately predicts foot ulceration in a primary healthcare setting.

Conclusion

This study illustrates that the views and experiences of potential end users can be used effectively to develop and evaluate a diabetic foot risk tool with CDSS based on evidence-based guidelines integrated into an electronic health record. The risk tool integrates simple assessment readily available in a clinical setting, and should have minimal impact on experienced healthcare professionals’ usual workflow and reflects current Australian guidelines [24]. It also structures foot examination for those who are less proficient in diabetic foot assessment and will ensure that identified predictors of risk for foot ulcers are referred in a more timely way for definitive assessment and management.

References

  1. Australian Institute of Health and Welfare. Diabetes: Australian facts 2008. Australian Institute of Health and Welfare, Canberra. 2008. http://www.aihw.gov.au/publications/cvd/daf08/daf08.pdf. Accessed 25 March 2012.

  2. Pecoraro RE, Reiber GE, Burgess EM. Pathways to diabetic limb amputation. Basis for prevention. Diabetes Care. 1990;13(5):513–21.

    Article  CAS  PubMed  Google Scholar 

  3. Larsson J, Apelqvist J, Agardh CD, Stenstrom A. Decreasing incidence of major amputation in diabetic patients: a consequence of a multidisciplinary foot care team approach? Diabet Med. 1995;12(9):770–6.

    Article  CAS  PubMed  Google Scholar 

  4. Boulton AJ. The diabetic foot: from art to science. The 18th Camillo Golgi lecture. Diabetologia. 2004;47(8):1343–53. doi:10.1007/s00125-004-1463-y.

    Article  CAS  PubMed  Google Scholar 

  5. National Association of Diabetes Centres. Australian National Diabetes Information Audit & Benchmarking. Sydney. 2011. http://www.health.gov.au/internet/publications/publishing.nsf/Content/pq-diabetes-pubs-andiab11-toc. Accessed February 22, 2014.

  6. Wild S, Roglic G, Green A, Sicree R, King H. Global prevalence of diabetes: estimates for the year 2000 and projections for 2030. Diabetes Care. 2004;27(5):1047–53.

    Article  PubMed  Google Scholar 

  7. Boulton AJ, Armstrong DG, Albert SF, Frykberg RG, Hellman R, Kirkman MS, et al. Comprehensive foot examination and risk assessment: a report of the task force of the foot care interest group of the American Diabetes Association, with endorsement by the American Association of Clinical Endocrinologists. Diabetes Care. 2008;31(8):1679–85. doi:10.2337/dc08-9021.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Brownrigg JR, Apelqvist J, Bakker K, Schaper NC, Hinchliffe RJ. Evidence-based management of PAD & the diabetic foot. Eur J Vasc Endovasc Surg. 2013;45(6):673–81. doi:10.1016/j.ejvs.2013.02.014.

    Article  CAS  PubMed  Google Scholar 

  9. Bakker K, Schaper NC, on behalf of the International Working Group on the Diabetic Foot Editorial Board. The development of global consensus guidelines on the management and prevention of the diabetic foot 2011. Diabetes Metab Res Rev. 2012;28(S1):116-8. doi:10.1002/dmrr.2254.

  10. Crawford F, Inkster M, Kleijnen J, Fahey T. Predicting foot ulcers in patients with diabetes: a systematic review and meta-analysis. QJM. 2007;100(2):65–86. doi:10.1093/qjmed/hcl140.

    Article  CAS  PubMed  Google Scholar 

  11. Litzelman DK, Marriott DJ, Vinicor F. Independent physiological predictors of foot lesions in patients with NIDDM. Diabetes Care. 1997;20(8):1273–8.

    Article  CAS  PubMed  Google Scholar 

  12. Bergin SM, Brand CA, Colman PG, Campbell DA. The impact of socio-economic disadvantage on rates of hospital separations for diabetes-related foot disease in Victoria, Australia. J Foot Ankle Res. 2011;4:17. doi:10.1186/1757-1146-4-17.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Leese GP, Feng Z, Leese RM, Dibben C, Emslie-Smith A. Impact of health-care accessibility and social deprivation on diabetes related foot disease. Diabet Med. 2013;30(4):484–90. doi:10.1111/dme.12108.

    Article  CAS  PubMed  Google Scholar 

  14. McNeely MJ, Boyko EJ, Ahroni JH, Stensel VL, Reiber GE, Smith DG, et al. The independent contributions of diabetic neuropathy and vasculopathy in foot ulceration. How great are the risks? Diabetes Care. 1995;18(2):216–9.

    Article  CAS  PubMed  Google Scholar 

  15. Sriussadaporn S, Mekanandha P, Vannasaeng S, Nitiyanant W, Komoltri C, Ploybutr S, et al. Factors associated with diabetic foot ulceration in Thailand: a case-control study. Diabet Med. 1997;14(1):50–6. doi:10.1002/(SICI)1096-9136(199701)14:1<50::AID-DIA292>3.0.CO;2-6.

    Article  CAS  PubMed  Google Scholar 

  16. Boyko EJ, Ahroni JH, Cohen V, Nelson KM, Heagerty PJ. Prediction of diabetic foot ulcer occurrence using commonly available clinical information: the Seattle Diabetic Foot Study. Diabetes Care. 2006;29(6):1202–7. doi:10.2337/dc05-2031.

    Article  PubMed  Google Scholar 

  17. Rith-Najarian SJ, Stolusky T, Gohdes DM. Identifying diabetic patients at high risk for lower-extremity amputation in a primary health care setting. A prospective evaluation of simple screening criteria. Diabetes Care. 1992;15(10):1386–9.

    Article  CAS  PubMed  Google Scholar 

  18. Mayfield JA, Reiber GE, Nelson RG, Greene T. A foot risk classification system to predict diabetic amputation in Pima Indians. Diabetes Care. 1996;19(7):704–9.

    Article  CAS  PubMed  Google Scholar 

  19. Lavery LA, Armstrong DG, Vela SA, Quebedeaux TL, Fleischli JG. Practical criteria for screening patients at high risk for diabetic foot ulceration. Arch Intern Med. 1998;158(2):157–62. doi:10.1001/archinte.158.2.157.

    Article  CAS  PubMed  Google Scholar 

  20. Peters EJ, Lavery LA, International Working Group on the Diabetic F. Effectiveness of the diabetic foot risk classification system of the International Working Group on the Diabetic Foot. Diabetes Care. 2001;24(8):1442–7.

    Article  CAS  PubMed  Google Scholar 

  21. Lavery LA, Peters EJ, Williams JR, Murdoch DP, Hudson A, Lavery DC, et al. Reevaluating the way we classify the diabetic foot: restructuring the diabetic foot risk classification system of the International Working Group on the Diabetic Foot. Diabetes Care. 2008;31(1):154–6. doi:10.2337/dc07-1302.

    Article  PubMed  Google Scholar 

  22. Leese GP, Reid F, Green V, McAlpine R, Cunningham S, Emslie-Smith AM, et al. Stratification of foot ulcer risk in patients with diabetes: a population-based study. Int J Clin Pract. 2006;60(5):541–5. doi:10.1111/j.1368-5031.2006.00899.x.

    Article  CAS  PubMed  Google Scholar 

  23. National Institute for Health and Clinical Excellence. Diabetic foot problems: prevention and management National Institute for Health and Clinical Excellence, London. 2015. http://www.nice.org.uk/guidance/ng19. Accessed 22 Sept 2015.

  24. National Health and Medical Research Council. National Evidence-Based Guideline: Prevention, Identification and Management of Foot Complications in Diabetes. (Part of the Guidelines on the management of Type 2 Diabetes). National Health and Medical Research Council, Melbourne. 2011. http://t2dgr.bakeridi.edu.au/LinkClick.aspx?fileticket=Z_vePysE8Ng%3D&tabid=760&mid=1629. Accessed 10 Apr 2011.

  25. Scottish Intercollegiate Guidelines Network. Management of diabetes A national clinical guideline. Scottish Intercollegiate Guidelines Network, Edinburgh. 2010. http://www.sign.ac.uk/pdf/sign116.pdf. Accessed 5 Apr 2011.

  26. Abbott CA, Carrington AL, Ashe H, Bath S, Every LC, Griffiths J, et al. The North-West Diabetes Foot Care Study: incidence of, and risk factors for, new diabetic foot ulceration in a community-based patient cohort. Diabet Med. 2002;19(5):377–84.

    Article  CAS  PubMed  Google Scholar 

  27. Monteiro-Soares M, Vaz-Carneiro A, Sampaio S, Dinis-Ribeiro M. Validation and comparison of currently available stratification systems for patients with diabetes by risk of foot ulcer development. Eur J Endocrinol. 2012;167(3):401–7. doi:10.1530/EJE-12-0279.

    Article  CAS  PubMed  Google Scholar 

  28. Crawford F, Cezard G, Chappell FM, Murray GD, Price JF, Sheikh A, et al. A systematic review and individual patient data meta-analysis of prognostic factors for foot ulceration in people with diabetes: the international research collaboration for the prediction of diabetic foot ulcerations (PODUS). Health Technol Assess. 2015;19(57):1–210. doi:10.3310/hta19570.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Bower VM, Hobbs M. Validation of the basic foot screening checklist: a population screening tool for identifying foot ulcer risk in people with diabetes mellitus. J Am Podiatr Med Assoc. 2009;99(4):339–47.

    Article  PubMed  Google Scholar 

  30. Shortliffe EH, Davis R, Axline SG, Buchanan BG, Green CC, Cohen SN. Computer-based consultations in clinical therapeutics: explanation and rule acquisition capabilities of the MYCIN system. Comput Biomed Res. 1975;8(4):303–20.

    Article  CAS  PubMed  Google Scholar 

  31. Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health. 2014;104(12):e12–22. doi:10.2105/AJPH.2014.302164.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi:10.1136/bmj.38398.500764.8F.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Durkin J. Expert systems:Design and Development. London: Englewood Cliffs; 1994.

    Google Scholar 

  34. Department of Health Western Australia. High Risk Foot Model of Care. Health Networks Branch, Perth. 2010. http://www.healthnetworks.health.wa.gov.au/modelsofcare/docs/High_Risk_Foot_Model_of_Care.pdf. Accessed 17 Jun 2010.

  35. Department of Health Western Australia. Diabetes in Western Australia: Prevalence and services in 2012. Health Networks Branch, Perth. 2012. http://www.healthnetworks.health.wa.gov.au/docs/WA_diabetes_services_review_Report.pdf. Accessed 16 Apr 2013.

  36. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120(2):135–42.

    Article  CAS  PubMed  Google Scholar 

  37. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339–46.

    Article  CAS  PubMed  Google Scholar 

  38. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38. doi:10.1001/jama.293.10.1223.

    Article  CAS  PubMed  Google Scholar 

  39. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29–43. doi:10.7326/0003-4819-157-1-201207030-00450.

    Article  PubMed  Google Scholar 

  40. Roshanov PS, Misra S, Gerstein HC, Garg AX, Sebaldt RJ, Mackay JA, et al. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011;6:92. doi:10.1186/1748-5908-6-92.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. doi:10.1136/bmj.f657.

    Article  PubMed  Google Scholar 

  42. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):iii–iv. 1-72.

    Article  CAS  PubMed  Google Scholar 

  43. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362(9391):1225–30. doi:10.1016/S0140-6736(03)14546-1.

    Article  PubMed  Google Scholar 

  44. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999;6(2):104–14.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Nilasena DS, Lincoln MJ. A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines. Proc Annu Symp Comput Appl Med Care. 1995;640:5.

    Google Scholar 

  46. Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. Am J Med. 1997;102(1):89–98.

    Article  CAS  PubMed  Google Scholar 

  47. Montori VM, Dinneen SF, Gorman CA, Zimmerman BR, Rizza RA, Bjornsen SS, et al. The impact of planned care and a diabetes electronic management system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. Diabetes Care. 2002;25(11):1952–7.

    Article  PubMed  Google Scholar 

  48. Meigs JB, Cagliero E, Dubey A, Murphy-Sheehy P, Gildesgame C, Chueh H, et al. A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project. Diabetes Care. 2003;26(3):750–7.

    Article  PubMed  Google Scholar 

  49. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, et al. Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA. 2000;284(11):1411–6.

    Article  CAS  PubMed  Google Scholar 

  50. Khoury A, Landers P, Roth M, Rowe N, DaMert G, Dahar W, et al. Computer-supported identification and intervention for diabetic patients at risk for amputation. MD Computing. 1998;15(5):307–10.

    CAS  PubMed  Google Scholar 

  51. Peterson KA, Radosevich DM, O’Connor PJ, Nyman JA, Prineas RJ, Smith SA, et al. Improving Diabetes Care in Practice: findings from the TRANSLATE trial. Diabetes Care. 2008;31(12):2238–43. doi:10.2337/dc08-2034.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Crawford F, Bekker HL, Young M, Sheikh A. General practitioners’ and nurses’ experiences of using computerised decision support in screening for diabetic foot disease: implementing Scottish Clinical Information - Diabetes Care in routine clinical practice. Inform Prim Care. 2010;18(4):259–68.

    PubMed  Google Scholar 

  53. Carroll C, Marsden P, Soden P, Naylor E, New J, Dornan T. Involving users in the design and usability evaluation of a clinical decision support system. Comput Methods Programs Biomed. 2002;69(2):123–35.

    Article  PubMed  Google Scholar 

  54. Gould JD, Boies SJ, Lewis C. Making Usable, Useful, Productivity - Enhancing Computer-Applications. Commun ACM. 1991;34(1):74–85. doi:10.1145/99977.99993.

    Article  Google Scholar 

  55. MMEx eHealth Platform. Perth. 2007. http://www.mmex.net.au/Products.aspx. Accessed 11 March, 2011.

  56. Tashakkori A, Teddlie C. Handbook of mixed methods in social & behavioral research. Thousand Oaks: SAGE Publications; 2003.

    Google Scholar 

  57. Ammenwerth E, Iller C, Mansmann U. Can evaluation studies benefit from triangulation? A case study. Int J Med Inform. 2003;70(2-3):237–48. doi:10.1016/s1386-5056(03)00059-5.

    Article  PubMed  Google Scholar 

  58. Kaplan B. Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform. 2001;64(1):39–56. doi:10.1016/S1386-5056(01)00184-8.

    Article  CAS  PubMed  Google Scholar 

  59. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med. 2009;6(8):e1000126. doi:10.1371/journal.pmed.1000126.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Wyatt J. Quantitative evaluation of clinical software, exemplified by decision support systems. Int J Med Inform. 1997;47(3):165–73.

    Article  CAS  PubMed  Google Scholar 

  61. Kaplan B, Duchon D. Combining Qualitative and Quantitative Methods in Information-Systems Research - a Case-Study. MIS Q. 1988;12(4):571–86. doi:10.2307/249133.

    Article  Google Scholar 

  62. Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med Inform (Lond). 1990;15(3):205–17.

    Article  CAS  Google Scholar 

  63. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8(6):527–34.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Kilsdonk E, Peute LW, Riezebos RJ, Kremer LC, Jaspers MW. From an expert-driven paper guideline to a user-centred decision support system: a usability comparison study. Artif Intell Med. 2013;59(1):5–13. doi:10.1016/j.artmed.2013.04.004.

    Article  PubMed  Google Scholar 

  65. Kaplan B. Evaluating informatics applications--clinical decision support systems literature review. Int J Med Inform. 2001;64(1):15–37. doi:10.1016/S1386-5056(01)00183-6.

    Article  CAS  PubMed  Google Scholar 

  66. Nielsen J. Applying Discount Usability Engineering. Ieee Softw. 1995;12(1):98–100. doi:10.1109/52.363161.

    Article  Google Scholar 

  67. Leslie SJ, Hartswood M, Meurig C, McKee SP, Slack R, Procter R, et al. Clinical decision support software for management of chronic heart failure: development and evaluation. Comput Biol Med. 2006;36(5):495–506. doi:10.1016/j.compbiomed.2005.02.002.

    Article  PubMed  Google Scholar 

  68. Curran MJ, Campbell J, Rugg G. An investigation into the clinical reasoning of both expert and novice podiatrists. Foot. 2006;16(1):28–32. doi:10.1016/j.foot.2005.11.001.

    Article  Google Scholar 

  69. Edelman D, Sanders LJ, Pogach L. Reproducibility and accuracy among primary care providers of a screening examination for foot ulcer risk among diabetic patients. Prev Med. 1998;27(2):274–8. doi:10.1006/pmed.1998.0263.

    Article  CAS  PubMed  Google Scholar 

  70. Leese GP, Cochrane L, Mackie AD, Stang D, Brown K, Green V. Measuring the accuracy of different ways to identify the ‘at-risk’ foot in routine clinical practice. Diabet Med. 2011;28(6):747–54. doi:10.1111/j.1464-5491.2011.03297.x.

    Article  CAS  PubMed  Google Scholar 

  71. National Health and Medical Research Council. Prevention, Identification and Management of Foot Complications in Diabetes Technical Report. National Health and Medical Research Council, Melbourne 2011. http://t2dgr.bakeridi.edu.au/LinkClick.aspx?fileticket=IGRCbi9Seqs%3d&tabid=172. Accessed 10 Jun 2011.

Download references

Author information

Authors and Affiliations

  1. Western Australian Centre for Rural Health, Faculty of Medicine, Dentistry & Health Sciences, The University of Western Australia, M706, 35 Stirling Highway, Crawley, 6009, WA, Australia

    Deborah E. Schoen

  2. Director Centre for Software Practice, The University of Western Australia, M002, 35 Stirling Highway, Crawley, 6009, WA, Australia

    David G. Glance

  3. Director Western Australian Centre for Rural Health, Faculty of Medicine, Dentistry & Health Sciences, The University of Western Australia, M702, 35 Stirling Highway, Crawley, 6009, WA, Australia

    Sandra C. Thompson

Authors
  1. Deborah E. Schoen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David G. Glance
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sandra C. Thompson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deborah E. Schoen.

Additional information

Competing interests

The authors declare they have no competing interests.

Authors’ contributions

DES and DGG conceived the computerised clinical decision support system, DES researched data, wrote manuscript and reviewed/edited manuscript. DDG and SCT contributed to the discussion and reviewed/edited manuscript. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schoen, D.E., Glance, D.G. & Thompson, S.C. Clinical decision support software for diabetic foot risk stratification: development and formative evaluation. J Foot Ankle Res 8, 73 (2015). https://doi.org/10.1186/s13047-015-0128-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13047-015-0128-z

Keywords

Journal of Foot and Ankle Research

ISSN: 1757-1146