KPMG report on AI benefits reportedly riddled with AI hallucinations
A report published by Big Four accounting firm KPMG about the benefits of agentic AI has been found to contain numerous fake citations and inaccurate claims, likely generated by AI.
In October last year, KPMG, one of the world's Big Four professional services and accounting firms, published a report titled "Total Experience: Redefining Excellence in the Age of Agentic AI." The document discussed how companies are using AI to cater to customers' needs. However, an investigation by GPTZero, a creator of AI content detection tools, and verified by the Financial Times, revealed that the report was full of inaccuracies and fake footnotes.
GPTZero found that only five out of 45 citations in the report accurately pointed to real sources. Twenty-eight citations paraphrased titles or added fake components to real sources, while twelve were too vague to determine their existence. GPTZero coined the term "vibe citing" for this phenomenon.
In addition to fake citations, roughly half of the claims in the paper were either fake or misattributed. GPTZero stated these were "likely the result of an AI research tool over-complying with a request to find examples of 'agentic AI' in the wild."
Notable examples include KPMG claiming that Emirates launched a mobile chatbot called Sara capable of conversing with passengers and altering their flights. In reality, Sara is a mobile assistant launched in 2023 that is not an AI-powered chatbot and cannot change bookings. KPMG also claimed that Swiss bank UBS integrated agentic AI across investment advisory, risk management, and compliance monitoring. UBS told the Times this was "factually incorrect." Another example involved Swiss Federal Railways (SBB), which KPMG said had AI agents to help passengers plan and optimize trips. An SBB spokesperson called this "not accurate."
Reports from firms like KPMG are typically considered highly trustworthy and are often cited in other research. GPTZero CEO Edward Tian warned that error-ridden papers from the Big Four could "poison the well of information" and lead to second-hand AI hallucinations. A KPMG spokesperson told the Times that the company "takes the accuracy and integrity of its published content seriously." KPMG has since retracted the report and is reviewing the circumstances surrounding its publication.
