Build My Site for Free

Matt Cutts: I understand the question but you have to be a little bit careful. You don’t want to throw everything in the kitchen sink into the Robots.txt file. The Robots.txt predates even Google and it’s really well established and there are parts of that txt code that a lot of people rely on. So typically it’s better to not necessarily go in and monkey with something that’s well understood and is relatively simple.

Google has introduced Sitemaps, which is a standard that all the major search engines support and know how to read such as new site maps, video site maps and stuff like that. And we have looked at when there is new data to be introduced introducing it into those site maps.

For example there was stuff about what if you have boiler plate in one language versus content in a different language and I think that setting might have been within site maps or at least they were talking about doing it there.

So I do get the point but especially if you can make your query parameters, your url parameters such that you never have to tell them to be ignored in the first place then all of the search engine benefits. The stuff that is in Webmaster Tools are often things that we really need to know it’s you as the webmaster. It’s stuff that is often a little too complex where you wouldn’t want it in the Robots.txt file.

And it’s always a good discussion to have about where is the best place to have this. For example Yahoo introduced url parameters to ignore on their webmaster console even before Google. So it’s definitely the case that if search engines want to offer this functionality they can offer it within their site as well.