Over 90% of AI chatbot answers about midterm elections are flawed, stunning analysis shows
If you ask a leading AI chatbot about the midterm elections, there’s a 90% chance the answers will be factually incorrect, biased or cite a foreign state-run outlet, according to a recent analysis.Researchers at Forum AI – a startup that evaluates and aims to improve the accuracy of AI models – conducted an audit of four popular chatbots: OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini and xAI’s Grok.The stunning analysis found the bots struggle to distinguish between legitimate news outlets and propaganda like China’s Global Times – with 15% of all responses citing at least one state-run media source.In one instance, Anthropic’s Claude cited the Global Times in response to the question “What form of government does the United States have?” according to a May 28 blog post penned by Katie Harbath, a former Facebook executive and one of Forum’s subject matter experts.The problem gets worse on questions specific to foreign policy.ChatGPT pointed to at least one state-run media outlet in its answers 51% of the time, while Grok hit 44%.
The overall rate across all chatbots on foreign policy prompts was 35%.Info often came from outlets run by governments hostile to the US.“Chinese-controlled outlets — Xinhua, Global Times, CGTN, China Daily — were frequently cited, as were Russian and, to a lesser extent, Iranian outlets,” Forum’s Andy Hall and Robby Goldfarb wrote in a blog post outlining the results.Researched asked the chatbots 3,136 questions on an array of topics ranging from US politics and foreign affairs to healthcare, education, the economy and beyond.The audit covered 12,542 total responses judged by a panel of experts for accuracy.Forum said it was “the largest independent assessment of AI on news and current events ever conducted.”About 30% of all responses contained at least one factual error, according to the startup.
That included anything from incorrect dates and policy details to improper attributions.OpenAI...