Jill Whalen’s post about Getting into Google was terrific, full of lots of great information on robots.txt, crawling, sitemaps and the dreaded Google Supplemental Index.
One of the points Google’s Dan Crow made that stuck out for me was about getting pages crawled that weren’t linked from anywhere else:
He said that you could simply submit one of the URLs via your sitemap, and then they’d crawl the rest. He also talked about how sitemaps were good for getting pages indexed that could be reached only via webforms. He did admit later that even though those pages would be likely to be indexed via the sitemap, at this time they would still most likely be considered low quality since they wouldn’t have any PageRank. Google is working on a way to change this in the future, however.
This, of course, means that even when you submit a sitemap file to Google to get the problem pages crawled, the problem pages won’t have Page Rank. If they had Page Rank, that means they’d have backlinks, which would probably get them crawled, so they will in all likelyhood wind up in the Google Supplemental Index.
He explained that basically the supplemental index is where they put pages that have low PageRank (the real kind) or ones that don’t change very often. These pages generally don’t show up in the search results unless there are not enough relevant pages in the main results to show.
Uh, isn’t that kind of a Catch 22 – you can get them crawled but condemned to Google Hell?
At least, that’s how it is at the moment. Jill also reported that Dan indicated that the Supplemental Index might be on the way out (YES!) as Google is beginning to crawl the Supplemental Index more often, which means the line between regular and “supplemental” results will blur a bit.
But (and this is a BIG but), without good SEO, backlinks and Page Rank, those pages might as well still be in the Supplemental Results. Showing up at position 1,234,987 for a search in the regular results is still, well, Google Hell.
So, everything that has been written up until now about getting out of the Google Supplemental Index will still be applicable if or when Google does actually remove the supplemental index.
The moral? Stop crying and get to work adding good, quality content and backlinks to those “problem” pages.