Artificial intelligence is increasingly being applied in critical applications such as healthcare, recruitment, and even self-driving cars. Given that the decisions and recommendations made by these systems can have a significant impact on an individual’s life — recruitment algorithms on someone’s career, healthcare algorithms on diagnosis and treatment, and self-driving cars on the driver, passengers and others’ safety — it is more important than ever that the algorithms behind these systems are efficacious. In other words, algorithms must do what they are designed to do to an acceptable level to avoid harm, internally or externally. Indeed, when a process is fully automated, the system’s failure can be catastrophic, as was seen with the failure of Knight Capital’s trading algorithm, which cost the company over $440 million.

Typically, the ground truth or actual values are compared to the values predicted by the model. For example, in the case of recruitment, this could be a hiring manager’s judgements on the hireability of a candidate. The way that efficacy is measured depends on the type of system and its output, with different approaches being suitable for regression and classification systems. In this blog post, we give an overview of some methods for measuring algorithm efficacy.