Google SEO Tip #5 – Can I use Robots.txt to Optimize Google’s Crawl?
Matt Cutts: This is another one of those NOOO kind of videos. I swear I had completely brown hair until you asked this question and then suddenly gray just popped in like that. That’s where the gray came from, really.
So no, please don’t use robots.txt in an attempt to sort of shunt Googlebot all the way over to one section of a website, but only for a week. Although we try to fetch robots.txt on a sort of daily basis or once every few hundred fetches to make sure we have an accurate copy of robots.txt weird things can happen if you’re trying to flail around and change your robots.txt really fast.
The other thing is that’s really not the best mechanism to handle it. Robots.text is not the best way to do that. Suppose you want to make sure a section of ten pages gets crawled well, it’s much better to take those ten pages and link to them from your root page and say, “Hey, our featured category this week is red widgets instead of brown widgets or blue widgets.” And then just link to all ten red widget pages.
That’s because when all the PageRank comes into the root page of your site, which is where most of your PageRank typically comes in because most people typically link to the root of your website. If you put the links to the pages that you care about right up front and center on that root page, then PageRank flows more so to those pages than to the rest of the pages on your site that might be five, six or seven links away from your root page.
What I would say is you could try using robots.txt. I really don’t think it would work. You would be much more likely to shoot yourself in the foot by trying to jump around and swap out different robots.txt every week. What’s much better is instead to work on your site architecture to re-architect things such that the pages you want to highlight and would like more PageRank and more crawling are linked more directly or more closely from your root page. That will lead Googlebot more into that part of your site.
So, please, don’t try to just swap in and out different robots.txt and sort of say, “Ok, now you’re going to crawl to this part of the site this week and this part of the site next week.” You’re much more likely to just confuse Googlebot and Googlebot might say, “You know what, maybe I just won’t crawl any of these pages. This seems strange to me.”
So, that’s the other way that I’d recommend is to change your site architecture and put it right at the front to make your site more crawlable that way.
