Recent research released by the BBC raises critical concerns regarding the reliability of AI assistants in providing news answers. The study highlights troubling instances of factual inaccuracies and improper representation of source material within these AI systems.

The results of the research are alarming and reveal the following insights:

• Over half (51%) of the responses generated by AI regarding news topics were found to have notable issues.
• Nearly one-fifth (19%) of AI-generated answers that referenced BBC content contained factual inaccuracies, including wrong statements, figures, and dates.
• Thirteen percent of the quotes attributed to BBC articles were either modified or did not actually exist in the referenced text.

This study, conducted over a month, evaluated four leading AI assistants—OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity. These systems were granted access to the BBC’s website and tasked with answering news-related queries using BBC News articles as references. BBC journalists, seasoned experts on the topics, reviewed the AI responses based on criteria such as accuracy, neutrality, and adherence to BBC content.

According to Pete Archer, Programme Director for Generative AI at the BBC, “We are enthusiastic about the potential of AI and its ability to enhance audience experiences. We have already integrated AI technologies for subtitling on BBC Sounds and translating content on BBC News. When utilized responsibly, AI can add significant value.”

“However, AI also presents substantial challenges for audiences. While individuals may feel they can rely on the information provided by these AI assistants, this research demonstrates that their responses concerning important news events can be distorted, misleading, or factually incorrect. As the use of AI assistants expands, ensuring the information they deliver is accurate and reliable is paramount.”

“Media outlets like the BBC should maintain authority over how their content is utilized, while AI firms must clarify how these assistants process news, along with the extent of inaccuracies they might generate. Achieving this necessitates strong collaborations between AI and media organizations, fostering innovative methods that prioritize audience needs and enhance value for all parties involved. The BBC is eager to explore partnerships that will achieve these goals.”

Examples of significant issues noted in responses from these AI assistants include:

• Both ChatGPT and Copilot asserted that former Prime Minister Rishi Sunak and former First Minister Nicola Sturgeon were still in office after their departures.
• Gemini incorrectly claimed that “The NHS advises against starting to vape and suggests that smokers seeking to quit should consider other options.” In reality, the NHS does endorse vaping as a smoking cessation method.
• A response from Perplexity regarding the escalating conflict in the Middle East cited BBC as the source, inaccurately stating that Iran initially showed ‘restraint’ and described Israel’s actions as ‘aggressive’—terms not used in the BBC’s impartial reporting.

The complete research findings can be accessed on the BBC website.

For further reading: Insight from Deborah Turness, CEO, BBC News and Current Affairs

IW