Can AI be relied upon to run mission-critical systems and make judgement calls?

Concerns are on the rise about reliability in AI predictions. Human intuition still beats AI hands down in making judgment calls in a crisis. People – especially those working in their areas of expertise – are simply more trustworthy. Studies have shown that professionals such as air traffic controllers or nuclear power plant operators are highly reliable even in high-risk situations as, unlike AI systems, they can detect, contain and recover from errors, and practice improvisational problem-solving. While current AI systems are great at situational awareness, they are less good at anomaly detection, and improvising solutions. Can AI mature? asks Fred Werner, Head of Strategic Engagement, ITU Telecommunication Standardisation Bureau.
Concerns are on the rise about reliability in AI predictions. Human intuition still beats AI hands down in making judgment calls in a crisis. People – especially those working in their areas of expertise – are simply more trustworthy. Studies have shown that professionals such as air traffic controllers or nuclear power plant operators are highly reliable even in high-risk situations as, unlike AI systems, they can detect, contain and recover from errors, and practice improvisational problem-solving. While current AI systems are great at situational awareness, they are less good at anomaly detection, and improvising solutions. Can AI mature? asks Fred Werner, Head of Strategic Engagement, ITU Telecommunication Standardisation Bureau.

As artificial systems (AI) get increasingly complex, they are being used to make forecasts – or rather generate predictive model results – in more and more areas of our lives. But, at the same time, concerns are on the rise about reliability, amid widening margins of error in elaborate AI predictions. 

How can we address these concerns? 

Management science offers a set of tools that can make AI systems more trustworthy, according to Thomas G Dietterich, Professor Emeritus and Director of Intelligent Systems Research at Oregon State University. During a webinar on the AI for Good platform hosted by the International Telecommunication Union (ITU), he told our audience that the discipline that brings human decision-makers to the top of their game can also be applied to machines

Why is this important? Because human intuition still beats AI hands down in making judgment calls in a crisis. People – and especially those working in their areas of experience and expertise – are simply more trustworthy.  Studies by the University of California (UC), Berkeley, scholars Todd LaPorte, Gene Rochlin and Karlene Roberts found that certain groups of professionals, such as air traffic controllers or nuclear power plant operators, are highly reliable even in a high-risk situation.

These professionals develop a capability to detect, contain and recover from errors, and practice improvisational problem solving, said Dietterich. This is because of their “preoccupation with failure”. They are constantly watching for anomalies and near misses – and treating those as symptoms of a potential failure mode in the system.  Anomalies and near misses, rather than being brushed aside, are then studied for possible explanations, normally by a diverse team with wide-ranging specializations. Human professionals bring far higher levels of “situational awareness” and know when to defer to each other’s expertise.

These principles are useful when thinking about how to build an entirely autonomous and reliable AI system, or how to design ways for human organizations and AI systems to work together.  AI systems can acquire high situational awareness, thanks to their ability to integrate data from multiple sources and continually re-assess risks. However current AI systems, while adept at situational awareness, are less effective at anomaly detection and unable to explain anomalies and improvise solutions.

More research is needed before an AI system can reliably identify and explain near-misses.  We have systems that can diagnose known failures, but how do we diagnose unknown failures? What would it mean for an AI system to engage in improvisational problem solving that somehow can extend the space of possibilities beyond the initial problem that the system was programmed to solve?

Where AI systems and humans collaborate, a shared mental model is needed. AI should not bombard its human counterparts with irrelevant information, and must also understand and be able to predict the behaviour of human teams. 

One way to train machines to explain anomalies, or to deal with spontaneity, could be exposure to the performing arts. Researchers and musicians at the Monash University in Melbourne and Goldsmiths University of London set out to explore whether AI  could perform as an improvising musician in a phantom jam session.  Mark d’Inverno, a jazz pianist and Professor of Computer Science at Goldsmiths in London, improvised live with Melbourne-based drummer and Monash University researcher Alon Ilsar. Completing the trio was an AI system, participating as a musician as well as an intermediary for the two artists who had never played together before.  During the session, the notes played on a MIDI piano in London by d’Inverno fed an algorithm, which modelled them to generate new notes in real-time and transmit them to Ilsar in Melbourne. Ilsar improvised in response with an AirSticks gestural instrument for electronic percussion.

Their goal was to emulate the real-life process of improvisation. Free-flowing, spontaneous improvisations are often considered the truest expression of creative artistic collaboration among musicians. ‘Jamming’ not only requires musical ability, but also trust, intuition and empathy towards one’s bandmates.

In the study, the first setting, called ‘Parrot’, repeats whatever is played. The second system autonomously plays notes regardless of a human musician’s contribution. The third also features complete autonomy, but counts the number of notes being played by the human musician to define the energy of the music. The fourth and most complicated system builds a mathematical model of the human artist’s music.  It listens carefully to what the musicians play and builds a statistical model of the notes, their patterns and even stores chord sequences.

Adding to this human/AI jamming session approach, Professor Dietterich see a further two promising approaches to improve, and mathematically “guarantee” trustworthiness. One is a competence model that can compute quantile regressions to predict AI behaviour, using the “conformal prediction” method to make additional corrections. Yet this approach requires lots of data and remains prone to misinterpretation.

The other way is to make autonomous systems deal with their “unknown unknowns” via open category detection. For instance, a self-driving car trained on European roads might have problems with kangaroos in Australia. An anomaly detector using unlabelled data could help the AI system respond more effectively to surprises.

 READ MORE:

As AI is deployed in more and more areas of our lives, what is becoming clear is that, far from a nightmare scenario of the machines taking over, the only way AI can be made more reliable, and more effective is for there to be a tighter than ever symbiosis between human systems and AI systems.  Only then can we truly rely on AI.

For more news from Top Business Tech, don’t forget to subscribe to our daily bulletin!

Follow us on LinkedIn and Twitter

Amber Donovan-Stevens

Amber is a Content Editor at Top Business Tech

Why ABM is Key to Strengthening your Marketing Strategy

Erin Lanahan • 16th May 2024

Account-Based Marketing (ABM) is revolutionizing B2B marketing by targeting high-value accounts with personalized strategies. Unlike traditional methods, ABM focuses on specific companies, delivering tailored content that meets their unique needs. This approach not only boosts ROI but also strengthens customer relationships and drives long-term growth. By aligning marketing and sales efforts, ABM ensures a unified...

Overcoming the Obstacles to AI Adoption

Kit Cox • 02nd May 2024

The power of AI combined with suitable use cases and a robust implementation plan can help businesses to radically reduce the time spent on manual, repetitive tasks, and allow teams to prioritise value-added work. But in all the excitement, it’s evident that many businesses are held back by inertia, and a lack of understanding about...

Overcoming the Obstacles to AI Adoption

Kit Cox • 02nd May 2024

The power of AI combined with suitable use cases and a robust implementation plan can help businesses to radically reduce the time spent on manual, repetitive tasks, and allow teams to prioritise value-added work. But in all the excitement, it’s evident that many businesses are held back by inertia, and a lack of understanding about...

How Predictive AI is Helping the Energy Sector

Colin Gault head of product at POWWR • 29th April 2024

In the past year or so, we have seen the emergence of many new and exciting applications for predictive AI in the energy industry to better maintain and optimise energy assets. In fact, the advances in the technology have been nothing short of rapid. The challenge, though, has been in supplying the ‘right’ data to...

How Predictive AI is Helping the Energy Sector

Colin Gault head of product at POWWR • 29th April 2024

In the past year or so, we have seen the emergence of many new and exciting applications for predictive AI in the energy industry to better maintain and optimise energy assets. In fact, the advances in the technology have been nothing short of rapid. The challenge, though, has been in supplying the ‘right’ data to...

Cheltenham MSP is first official local cyber advisor

Neil Smith Managing Director of ReformIT • 23rd April 2024

ReformIT, a Managed IT Service and Security provider (MSP) based in the UK’s cyber-capital, Cheltenham, has become the first MSP in the local area to be accredited as both a Cyber Advisor and a Cyber Essentials Certification Body. The Cyber Advisor scheme was launched by the Government’s official National Cyber Security Centre (NCSC) and the...

How we’re modernising BT’s UK Portfolio Businesses

Faisal Mahomed • 23rd April 2024

Nowhere is the move to a digitised society more pronounced than the evolution from the traditional phone box to our innovative digital street units. Payphone usage has dropped massively since the late 1990s/2000s, with devices and smart phones replacing not only communication access, but the central community points that the payphones once stood for. Our...

How we’re modernising BT’s UK Portfolio Businesses

Faisal Mahomed • 23rd April 2024

Nowhere is the move to a digitised society more pronounced than the evolution from the traditional phone box to our innovative digital street units. Payphone usage has dropped massively since the late 1990s/2000s, with devices and smart phones replacing not only communication access, but the central community points that the payphones once stood for. Our...