Date of Award
Master of Medical and Health Science by Research
School of Medical and Health Sciences
Professor Moira Sim
Dr Brennen Mills
The internet has impacted society and changed the way companies and individuals operate on a daily basis. Seeking information online via computer or mobile device is common practice. The phrase ‘Google it’ is now part of modern vernacular and is a resource increasingly utilised by young and old alike. Around 80% of Australian’s search health-related information online as it is convenient, cheap, and available 24/7. Symptom checkers are one tool used by consumers to investigate their health issues. Symptom checkers are automated online programs which use computerised algorithms, asking a series of questions to help determine a potential diagnosis and/or provide suitable triage advice. Recent evidence suggests symptom checkers may not work the way they are intended. Inferior or incorrect healthcare information can potentially have serious consequences on the consumer’s wellbeing and may not have the desired effect of directing consumers to the appropriate point of care.
This research evaluated the clinical performance of 36 symptom checkers found on websites and smartphone applications that are freely available for use by the Australian general public. Symptom checkers were exposed to 48 clinical vignettes, generating 1858 symptom checker vignette tests (SCVT). Diagnosis was assessed on the inclusion of the correct diagnosis in the first, the top three or top ten differential diagnoses (n = 1,170 SCVT). Triage advice was assessed on whether the triage category recommended was concordant with our assessment (n = 688 SCVT).
The correct diagnosis was listed first in 36% (95% CI 31–42) of SCVT, within the top three in 52% (95% CI 47–59) and within the top ten in 58% (95% CI 53–65). Symptom checkers which claimed to utilise artificial intelligence (AI) outperformed non-AI with the first listed diagnosis being accurate in 46% (95% CI 40–57) versus 32% (95% CI 26–38) of SCVT. Individual symptom checker performance varied considerably, with the average rate of correct diagnosis provided first ranging between 12%–-61%. Triage advice provided was concordant with our assessment in 49% (95% CI 44–54) of SCVT. Appropriate triage advice was provided more frequently for emergency care SCVT at 63% (95% CI 52–71) than for non-urgent SCVT at 30% (95% CI 11–39).
Symptom checker performance varied considerably in relation to diagnosis. Triage advice was risk-averse, typically recommending more urgent care pathways than necessary. Given this, symptom checkers may not be working to alleviate demand for health services (particularly emergency services) within Australia—counter to marketing materials of some organisations’ symptom checkers. It is important that symptom checkers do not further burden the healthcare system with inappropriate referrals or incorrect care advice. Although, a balance must be struck as avoiding unsuitable triage advice could potentially result in life-threatening consequences for consumers. Nonetheless, the results of this research make clear that the accuracy of diagnosis and triage advice provided from readily available symptom checkers for the Australian public require improvements before everyday consumers can rely entirely on health information provided via these mediums.
Hill, M. G. (2020). Appraisal of free online symptom checkers and applications for self-diagnosis and triage: An Australian evaluation. https://ro.ecu.edu.au/theses/2311