SEO Tip #136: Why Aren’t Geographic Target and Query Parameters Specified in Robots.txt

Matt Cutts: I understand the question but you have to be a little bit careful. You don’t want to throw everything in the kitchen sink into the Robots.txt file. The Robots.txt predates even Google and it’s really well established and there are parts of that txt code that a lot of people rely on. So typically it’s better to not necessarily go in and monkey with something that’s well understood and is relatively simple.

Google has introduced Sitemaps, which is a standard that all the major search engines support and know how to read such as new site maps, video site maps and stuff like that. And we have looked at when there is new data to be introduced introducing it into those site maps.

For example there was stuff about what if you have boiler plate in one language versus content in a different language and I think that setting might have been within site maps or at least they were talking about doing it there.

So I do get the point but especially if you can make your query parameters, your url parameters such that you never have to tell them to be ignored in the first place then all of the search engine benefits. The stuff that is in Webmaster Tools are often things that we really need to know it’s you as the webmaster. It’s stuff that is often a little too complex where you wouldn’t want it in the Robots.txt file.

And it’s always a good discussion to have about where is the best place to have this. For example Yahoo introduced url parameters to ignore on their webmaster console even before Google. So it’s definitely the case that if search engines want to offer this functionality they can offer it within their site as well.

Print Friendly, PDF & Email

About the Author

Andy Johnson

Andy Johnson has been on the Internet since the its dawn(ie his first computer program was recorded on cassette tape) and his first hard drive cost about as much his current MacBook. His first byline was in 1993 for a local newspaper rag he eventually helmed, and his last “real job” was at a computer start up which ended when it ended. Throughout it all he’s freelanced and blogged. Now he is mesmerized by Search Engine Optimization forever trying to “rise to the top” for the right reasons. He’s been married to his wife Julia for as long as he can remember and has two lovely, wonderful children. He looks forward to sharing the latest in the technical best for all the online entrepreneurs.