A map provides you with a lot of things besides a location. Maps show you the importance of a place, it’s connections to other place, and the distance between places.
A sitemaps doesn’t provide any of these things. It’s a simple list of URLs and when they were last updated. It’s more akin to an inventory than a map.
And whether you’ve submitted that inventory to Google or just let it hangout in the root folder of your website, there’s no guarantee that the URLs listed within it will be crawled or indexed by Google. In fact, a client of ours came to us after 95% of their URLs weren’t indexed despite being in their sitemap.
That’s because Google wants to make a real map—something that shows importance, connection, and distance. To do that, Google needs more than the list of URLs that your sitemaps offers. It needs to crawl the URLs of your website and traverse the links that connect them. It will integrate this information into the broad map of the web that it continuously builds and rebuilds.
So what does a sitemap actually do?
Despite not being a map, a sitemap is a useful thing. It’s your inventory of everything you think should be indexed by Google. Moreover, it’s the only vehicle by which you can get a list of URLs into Google Search Console—an invaluable tool that Google makes available to website owners—so you can check your list of URLs against Google.
So we shouldn’t think of a Sitemap as a solution to indexing problems. It’s not a way to get around having a site that can be quickly and easily crawled by Googlebot. Instead, we should think of sitemaps as a tool to help us diagnose indexing problems. It’s the beginning of any investigation into Google’s URL discovery process, not the end.
How do I know if I’m having indexation issues?
Check out the Google Search Console sitemaps indexation report by navigating to the Sitemaps menu, clicking on a sitemap file and then clicking on the “See Page Indexing” link at the top of the page:
From there, you’ll see the indexation report for an individual sitemap:
Depending on how your sitemaps are setup, this can allow you to narrow down problem by type or category of page. You’ll also be able to see a breakdown of indexation problems:
Sometimes these numbers are evidence of a problem. Other times, the pages being excluded for indexation are things you might expect, like category or tag pages that are purely navigation, which aren’t valuable content that Google would want to present to search users.
More to come on this topic in future posts.