An indexing study of 1.7 million pages identifies the biggest reason your pages are not indexed by Google: they are actively being removed from search results because of page quality.
Google is actively removing pages from its search results.
At Indexing Insight, we analysed the indexing data of 1.7 million pages across 18 websites. And found that 88% of not indexed pages were due to quality issues.
Important pages are actively being removed and forgotten by Google.
It doesn’t matter if you’re a large or small website. A big brand or a small brand. The trend is always the same. The biggest reason why your pages are not indexed is that they are actively being removed and forgotten by Google’s index.
Let’s dive into the methodology and findings.
The indexing data pulled in this study is from Indexing Insight. Here are a few more things to keep in mind when looking at the results:
Alright, let’s jump into the findings!
Based on our first-party data, this is what we found (and it surprised us!).
It doesn’t matter how we chop or slice the not indexed pages, the trend is always the same:
Quality indexing issues are the biggest reason why important pages are not indexed.
If we group all the not indexed pages across all the websites into the 3 types of indexing issues, we can see that quality issues make up 88% of all monitored indexing issues.
How do you define Page Quality?
The definition of Page Quality for us is Indexed pages being actively removed from Google’s search results. And Not Indexed pages actively being forgotten by the search index over time.
For more information, please read What is Page Quality?
This means that the biggest reason why important pages are not indexed is that they are being actively removed from Google’s search results and, over time, forgotten.
Let's break down the quality type by indexing state.
You can see that ‘URL is unknown to Google’ and ‘Discovered - currently not indexed’ make up 67% of the 500,000+ not indexed pages in the quality type.
These might appear to be crawling issues, but are in reality indexing issues.
Our research has found that ‘URL is unknown to Google’, ‘Crawled - currently not indexed’, and ‘Discovered - currently not indexed’ need new definitions.
Indexing states for Not Indexed pages can change over time as Google actively removes pages from search results.
Quality indexing issues are impacting different websites and brand sizes.
Even if we group the 3 types of not indexed pages by Moz brand authority, we get the same trend:
Quality indexing issues are the biggest reason why important pages are not indexed for big, medium or small brands.
If we group the 3 types of indexing issues by website type, we get the same trend for e-commerce, marketplace and listing websites. Quality indexing issues are the biggest reason important pages are not indexed.
Interestingly, news websites suffer more from technical indexing issues.
Finally, if we then group the 3 types of not indexed pages by website size, we can see the same trend: Quality indexing issues are the biggest reason why important pages are not indexed across both small and large websites.
The only exception to this rule is websites that are monitoring 100,000 - 500,000 pages. As most of the websites in this category are news websites.
The indexing data shows that marketplace and listing websites have the lowest Indexing Coverage score >70% (the % of pages indexed vs total monitored pages).
News websites had the best Index Coverage score for monitored pages at 97%.
Ecommerce websites didn’t have as many indexing issues, but still had less than 90% Index Coverage score for all the monitored pages.
Index Coverage Definition: The Index Coverage score is the % if indexed pages vs the total number of pages being monitored by Indexing Insight.
If we look at the raw indexed vs not indexed page numbers, we can see the scale of the indexing issues for marketplace and listing websites.
They make up a lot of the indexing issues we monitor.
If we layer Moz Brand Authority with our indexing data we can see that even big brands suffer from indexing issues for important traffic and revenue-driving pages.
The indexing data shows that both small and big brands have an Index Coverage score around 85 - 91%.
If we look at the raw data for indexed vs not indexed pages by Moz Brand Authority, we can see the scale of the problem for brands.
The indexing data shows that even big brands suffer from large-scale indexing issues.
Interestingly, if you only focus on ecommerce, marketplace and listing websites, the average Index Coverage score by Moz Brand Authority is lowered.
Big and small brands with Ecommerce, Marketplace and Listing websites suffer from much larger-scale indexing issues than news or blog websites.
This makes sense as these sorts of websites have lots more pages in our data set.
The findings from our indexing data were surprising.
Although we’ve seen Google actively remove pages, we were surprised to find the scale of quality issues across all monitored pages.
Our first-party data has uncovered that Google Search Console misreports indexing states for pages being actively forgotten. And that the definitions for ‘crawled - currently not indexed’ and ‘URL is unknown to Google’ need to change. But we never thought that quality issues would be the main reason that important traffic and revenue-driving pages are not indexed.
Especially for marketplace, listing and ecommerce websites with a big or small brand.
There was a core update on the 13th March, which could have contributed to the number of quality issues found in the monitored indexing data. We know that Google core updates impact on indexing. Still, this shows that there are quality issues on the website, which are causing pages to drop out of the index.
As Google continues to try to improve its core algorithms, we might see more important traffic and revenue-driving pages drop out of Google’s index.
Adam Gent
SEO Product Manager and Technical SEO. I’m currently an independent consultant who works with organisations to plan, scope and execute SEO projects that drive results.
