What do we do about AI overviews? What are they supposed to do?
Google says Google is doing the Googling for you.
What does that mean?
Rather than you searching, reading, and forming a conclusion in your mind or maybe citing things from sources in some bit of writing you’re doing, Google is creating a basic summary of information on a topic.
But how is Google doing that? How do these summaries get written? Where are the sources coming from and how?
If we know how this system works and how it cites sources, maybe we can hope to be one of those sources and get some traffic to our site.
We’ve gained some insight into how AI overviews work thanks to great work from Cyrus Sheperd.
Cyrus has shown that AI overviews can be very easily manipulated using spam techniques we thought died with the rollout of Panda and Penguin in the 2010s.
- By simply making his text match the AI overview, he was able to get his site into the AI overview footnotes
- This suggests that the LLM is writing the summary and then using Google search to find sources that match it’s conclusions
This is completely backwards. We should be looking for great sources and then drawing conclusions from them. Not backfilling conclusions from an LLM with whatever sources match most closely.
It conjures up visions of a nightmarish future, one where we’re incentivized to be more average. The web goes from a place where weird stuff and divergent opinions thrive to a sort of lingual equivalent of greige.
Lending further insight, Lily Ray VP of SEO at Amsive digital documented in a recent LinkedIn post that she’s run into 404, 410, and 405 codes in AI Overviews, suggesting that the right hand at Google doesn’t know what the left hand is doing.
Why would the world’s premier search company be recommending broken links to its users?
Lily also cites AI Overviews containing:
- Parasite SEO—when a marketer either buys or otherwise places content on a high-authority website, like a magazine or review site, in order to place links there and drive traffic to a sales page or some other commercial page.
- Random, 10-year-old posts on Reddit and Quora
- Indian sites with prices in rupees and no US shipping—this is significant because AI overviews is only available for US users and Google usually filters ecommerce results to show users things they can actually buy
- Lily also found links to extremely low quality blogs and ecommerce websites with little to no organic traction and lots of spammy AI-generated content
- Links to clearly biased sites, like plastic surgery sites recommending their own practice’s offering as the answer to a medical question
She sums up by saying that AI Overviews suggest that the future of search will be dead sites, low-quality sites, spammy AI content, and random international sites on searches that should be local, parked domains, and random Reddit or Quora posts.
What happened to favoring sites that show E-E-A-T (Experience, Expertise, Authority, and Trust) especially on YMYL (Your money, your life) content?
Why have I learned these acronyms and initialisms if Google isn’t going to use them?
It may be 2024, but AI Overviews are partying like it’s 2010—the world before Google Panda, when web spam still roamed free.
Nothing to See Here, Move Along
Despite Google CEO Sundar Pachai’s “this is all part of the plan” responses to criticisms brought up by The Verge and others, the criticism of AI overviews seems to be moving the needle as Google has reduced AI overviews from 27% to 11% of searches as tracked by BrightEdge, a search marketing firm.
The mistakes of AI Overviews are so prevalent that they’re now reported by mainstream news sites like The Washington Post and ABC News which both reported on search head Liz Reid’s statement about AI Overviews.
It’s AI All The Way Down
Oh, and we just found out that Google’s AI summaries are citing pages generated by the AI search engine Perplexity as sources. So the AI ouroboros is already here.
As Ryan Jones said on X, “An AI overview of an AI overview in search results pages in my search results page. Cool…..”
Yes, we have entered the second level of Inception where time slows down and search results become incredibly dumb.
Perplexity and RAG
Speaking of Perplexity, to me, it’s the competitor that demands a response from Google.
If you compare AI Overviews to Perplexity, Google’s new offering looks pretty slapdash. No wonder Perplexity’s CEO react to the Google I/O announcement with a laughing emoji.
The way Perplexity works is pretty simple. It’s search engine—so it matches a user’s query to documents found around the web—but then it asks its language model to summarize those sources with footnotes pointing to sites that backup its claims.
This method is called RAG—R-A-G—which stands for “retrieval-augmented generation.” IBM Research compares RAG system to a great writer and an excellent reference librarian working together—one finds the best resources, the other writes great prose to sum them up.
This approach is widely recognized in the AI field as one of the ways you solve the problem of LLMs hallucinating or otherwise providing unreliable information. Rather than asking the LLM to rely on its training data—the AI equivalent of just speaking off the top of your head—you ask it to read documents first and provide its answer based on those documents.
This is what I thought Google was going to build. It would have leaned into its unbeatable advantage in classic search, which is a form of information retrieval.
Yet it seems as though AI Overviews are drawing on training data—this would explain the 404s, parked domains, and foreign websites—and then using search where it can to paper over those results with relevant citations.
The Knowledge Graph
Google could also use its Knowledge Graph,Where PageRanks sees the world as a network of citations, the Knowledge Graph sees the world as a network of factual statements connecting things, like:
- New Dehli is a city in India.
- Harrison Ford starred in Indiana Jones.
- Emmanuel Macron is the President of France and resides at the Élysée Palace.
The Knowledge Graph as of 2023 contained over 1,500 billion such entities.
This is yet another way a computer system can understand, or at least model the world.
And that model could be used to fact-check and fine-tune AI-model. Again, Google could have leveraged an undeniable advantage.
Nah, Just Let Users Sort it Out
Instead of complementing LLMs with document retrieval or the Knowledge graph, Google seems to be hoping that user feedback will help them refine their AI.
Thanks to the DOJ suit against Google and the recent Google search leak, we know that Google relies on user interactions data—using a system it calls NavBoost—to refine search results. Presumably, Google is also collecting data on billions of interactions with AI Overviews.
Maybe this will work, but in the meantime Google’s damaging a brand built on the quality of its search results. Ands its doing so at a time when competition is more real than it’s been in over a decade, so much so that Google may already be losing market share.
The New New Thing
Ultimately we need to understand that while LLMs are impressive and powerful, they are just one way to model the world of information, and they don’t contain the same information as other models like PageRank or the Knowledge Graph.
An LLM has no way of judging the quality of the data in its training dataset and it has no real grasp of underlying facts. Maybe that will change and LLMs become more multi-model or are given corporeal form—many are saying foundation model robots are the path to AGI.
Until then, LLMs are a new and valuable tool, but they don’t supplant previous tools because previous tools contain unique information LLMs don’t.
Recommendations
Where does this leave us?
Right now, I’m not recommending that my clients change their approach to SEO because of AI. I’m certainly not going to recommend that anyone rewrite content to sound more like a machine in hopes of being cited by a flawed system that’s very likely to be revised in the near future.
So what can we do as publishers and SEOs to help Google craft better answers? We can lean into the things we know are applicable to all AI systems—they’re hungry for data and models can be improved through fine-tuning—which means using labeled data to refine the results.
This means the things we’re already doing, like making sites easier to crawl and labeling them with structured data, will help us feed LLMs and refine LLM responses. If we do enough of that, a future AI Overviews feature—one that works more like Perplexity—may use our understanding of whatever the subject matter may be to write the overview and therefore cite us as a source.
There’s a lot more to discuss here, like:
- Are AI overviews helpful? Are they good user interface design?
- Are LLMs hitting a performance ceiling?
- Where can we safely integrate LLMs into a publication process—and I’m not talking about content creation itself, but speeding up the rest of the process.
I’ll tackle these topics in future blogs posts and videos.