Technology
Audit finds over a third of Perplexity's numeric citations fail verification
A study testing factual queries across tech companies found 34.7% of citations attached to figures either lacked the cited numbers or could not be accessed.
The short version
- An audit evaluating Perplexity's search models on 310 factual questions found that 34.7% of citations attached to sentences with figures failed to contain those numbers or could not be opened.
- When evaluated at the claim level rather than per individual citation, 14.4% of 872 factual claims carrying figures failed verification.
- Around 16.1% of cited URLs were blocked behind logins or paywalls, while 25.1% had never been archived by the Wayback Machine.
- The audit authors noted the results represent a snapshot of English-language queries about tech companies and do not evaluate other sectors or languages.
Key facts
- In a test of 1,826 citations attached to figure-containing sentences, 34.7% pointed to pages that either failed to open or contained none of the cited figures.[Hacker News]
- Evaluated per claim, 14.4% of 872 factual claims containing numbers failed to find supporting figures across any of their cited links.[Hacker News]
- Out of 2,915 unique cited URLs evaluated for the sonar model, 16.1% were behind paywalls, logins, or bot walls, and 1.3% were dead links.[Hacker News]
- Perplexity's sonar and sonar-pro models passed on numeric verification at similar rates of 65.9% and 64.7%, respectively.[Hacker News]
- Of a sample of 1,432 resolving URLs cited by the model, 25.1% had never been indexed or archived by the Wayback Machine.[Hacker News]
What remains uncertain
- Whether cited gated pages actually support the claims remains unknown, as the audit only evaluated pages accessible without subscriptions or logins.[Hacker News]
- The study's scope was limited to a single snapshot of English-language queries about technology firms, leaving citation reliability in other topics and languages unmeasured.[Hacker News]