We’re both trying to figure out what users like and that’s the ideal sweet spot where we’re pursuing the exact same sort of things that you are trying to bake into your website. So, for example look at the things that people are complaining about. If people are complaining to Google that they don’t like content farms or article banks or whatever it is that they’re saying, “Oh Google I really don’t like this.”

We listen to that feedback as well and we try to figure out, ok if people really don’t like that how do we write an algorithm that’s going to return better content according to what our users are looking for? So we’re trying to suss out exactly what it is our users would like to have in the search results.

Maybe their complaining because one site is showing up to often in the top 10 and so they want more diversity, so we’ll write an algorithm to sort of say in our web search results, how do we encode more diversity?

So the thing to look for is try to get your finger on the pulse in terms of what users are unhappy about, what they want to see in Google, what they’re complaining about, because those are the exact same pieces of feedback that we’re listening to, even if we don’t talk about it outside of Google.

And so as a result if you’re able to assess, ok this is what Google is trying to change in order to be something that users like more, then you can provide those sorts of services or that functionality in your website. And that’s the sort of thing where now instead of chasing after the last algorithm and trying to respond to each individual update you’ll be sitting at the top of that mountain, hopefully whenever Googlebot crests, and we’ll find you right there.

The video cannot be shown at the moment. Please try again later.

Matt Cutts: Great question. In general we don’t promise that site: queries will rank in the exact same order that other pages would rank in. So we do use a few different factors.

We do use some version, roughly, of PageRank but it’s not exactly in PageRank order. We also look a little bit at, for example, maybe how short the URL is and those tend to be URLs at the root page or one directory down. And those tend to be the pages that would attract the most links anyway.

So it’s kind of a combination to sort of try and surface the pages we think are useful either according to PageRank or interesting in terms of being relatively short. So it’s something that’s pretty important or pretty close to your root page.

But I wouldn’t necessarily say that, and it’s not the case, that it’s strictly in PageRank order or anything like that, at least the last time that I checked. So it’s a relatively good proxy of the pages that might be kind of interesting but I wouldn’t treat it as a perfect list.

You can always go through your server logs and figure out which pages are driving the most traffic and sometimes those are going to be deep URLs, for example that might get a very specific link or that rank for some other reason.

Hope that helps.

The video cannot be shown at the moment. Please try again later.

The first one that I would do is try to include related questions so you have links from one question to another question. There is relatively good technology for this. Word Press for example has related post plug ins. There are several of those. Yet another related post and related things like that.

What that lets you do is as a user comes in they finish one question and they get to see four or five questions that are sort of similar to that. If they’re still interested in reading about that topic they can click on that and find the rest of your content.

It’s also good for Googlebot because Googlebot can come in and crawl one page and then find related pages via that crawling mechanism. So that way the way to get to a question on the fifth page doesn’t have to be going through page one and page two and page three and page four and finally clicking to page five to find that question. Instead they might click through to the first question and then find immediately a question on the fifth page that is related.

In general the more ways that people can sort of easily find content within the site, related is a great way to do it. That can lead to PageRank following within your site a little more easily sometimes.

The other thing to bear in mind is sometimes it sounded like you knew what questions were good. If you know what questions are good then you can highlight those. Now you can include things like voting buttons up and down so that you know which questions people really like. Or if you have things that are looking at your server logs to see what has generated a lot of traffic in the past.

Any way you have to know these are good questions you can highlight that for example on the root page. You could say, “These are our top 10 or our top 25 questions of all time.” Or make sure that those are questions that show up a little bit closer to the root so that the Googlebot doesn’t have to click through 20 different links in order to find those questions.

So those are a couple ways that you can think about to sort of make it so that your content is a little more surfaced, a little more discoverable, and then that way both users and Googlebot might be able to find your content a little more easily.

The video cannot be shown at the moment. Please try again later.

Definitely if we viewed domains as spammy if they had less than one year left that would be a ton of domains. So that’s not true. We don’t view anybody who doesn’t register three years at a time as some sort of spammy domain.

The thing is whenever you’re filing a patent, you want to make sure that you try to cover all of the implementations of that particular idea, but that doesn’t mean that that particular signal is used within search quality or within Google’s rankings.

So I happen to often renew my domains for two or three years at a time but I do it mainly for the convenience because I don’t ever want to lose mattcutts.com or any domain that’s important to me. So I wouldn’t really worry about this as a factor.

I know that there are at least a few domain name registrars that would love you to think, “register for 10 years and you’ll get a boost in search quality” and that’s not really the way it works. So consider all those factors. If you can renew it because it’s really easy and it’s convenient and you’re worried you might forget about it, fine go ahead. But I don’t think you are going to see yourself drop in the search results if you let yourself get down to less than a year left on your expiration date or anything like that.

Definitely just because something appears in a patent doesn’t mean that it’s necessarily used within a search engine or some other company’s results.
[yotube]DnfM_szmDh0[/youtube]

The fact is we go out of our way to try to prevent causing a lot of extra trouble during the Holidays. If there is some change that is a really huge impact and it’s right near Christmas or right near the Holidays sometimes we delay that into January to make sure the impact was not as big as otherwise.

So it’s definitely not our case to cause pain or stressed webmasters. What we want to do is return the best sites to users. At the same time we can’t just shut down for ¼ of the year any time in October, November or December and not make any changes to our search results. Because if we were only making changes ¾ of the time that would give plenty of opportunities for other search engines to keep making changes and keep improving.

So we have to keep making changes. We try to insulate webmasters where possible but it’s definitely not the case. I hate the idea of a Holiday update or that Google is always trying to make changes in order to make people buy ads or to cause uncertainty.

We do what we can within reasonable bounds to try and make sure that webmasters don’t have a huge amount of stress around the Holiday time. There is not much that we can do if we have a change that needs to go out so I’m not going to say we never change anything. But we do try to take reasonable steps in order to say, “Okay if that’s something that is really big that is optional and is right around the time where we could either delay it or do it early a lot of the times we’ll say, “Let’s hold off and do it in January instead.”

I hope that explains a little bit. I know that it’s very popular on some webmaster forums and websites for people to just say, “Google is trying to hurt me. Google is trying to hurt my business.” And that’s not the case. Our objective is always to return the right answers to users and if we can get that then we also try to do as much as we can for webmasters as well.

The video cannot be shown at the moment. Please try again later.

Whenever I decided to move back I used a 301 redirect and it took a period of several weeks because remember 301’s happen at a page level. So just because you see one 301 on one page of the old domain does not mean the entire domain has completely migrated.

What I did was set up the redirects such that every single page was redirecting from Dullest.com to MattCutts.com so it had been a complete transition. I really didn’t bother to check on Dullest.com for a few weeks, maybe a couple of months and then when I went back and looked at Google Analytics at that point all of my traffic had swapped over from Dullest.com to MattCutts.com.

So typically over a period of a few weeks, or several weeks, think about it like a couple months for example then we might be able to detect that a site has entirely moved. But if we’re getting mixed signals like some pages return a 200, which is an okay while other pages return a permanent or 301 redirect then we really don’t know what to make of that.

I’ve certainly seen some situations recently where a site had said I moved from olddomain.com to newdomain.com but they forgot to do a sub-domain and so they were still serving 200’s on the old sub-domain.

So it’s definitely not the case where you can assume everything will automatically and magically work perfectly. We do have a tool in Google Webmaster Tools where you can say, “My site has moved from here to here.” So you can do that, put the 301’s on each page level but I wouldn’t just assume it only has to be crawled once. Really Googlebot and Google need to build up enough confidence to really know that a site has fully migrated from the old site to the new site. So it can take a little while but hopefully after a while we do pick up on that.

The video cannot be shown at the moment. Please try again later.

Google has introduced Sitemaps, which is a standard that all the major search engines support and know how to read such as new site maps, video site maps and stuff like that. And we have looked at when there is new data to be introduced introducing it into those site maps.

For example there was stuff about what if you have boiler plate in one language versus content in a different language and I think that setting might have been within site maps or at least they were talking about doing it there.

So I do get the point but especially if you can make your query parameters, your url parameters such that you never have to tell them to be ignored in the first place then all of the search engine benefits. The stuff that is in Webmaster Tools are often things that we really need to know it’s you as the webmaster. It’s stuff that is often a little too complex where you wouldn’t want it in the Robots.txt file.

And it’s always a good discussion to have about where is the best place to have this. For example Yahoo introduced url parameters to ignore on their webmaster console even before Google. So it’s definitely the case that if search engines want to offer this functionality they can offer it within their site as well.

The video cannot be shown at the moment. Please try again later.

Your second question was, “Isn’t this Google trying to maximize its profit?” There has been a lot of times where Google has basically said yeah we could make money by selling some banner ad on our homepage but rather than annoy our users let’s try to figure out a way to get long-term loyalty from our users. The best way to do that is to protect them, help them and generate the kind of search results they are going to be happy to come for the next time around.

So there is definitely nothing in our search rankings where you will rank higher because you run AdSense. We try to take the long-term view and that would be a very short term amount of thinking to do so don’t worry about that within the search results.

The video cannot be shown at the moment. Please try again later.

If you want to go back into the very early days Larry and Sergei have these papers about the anatomy of a large scaled hyper text search engine, PageRank, you can find all of those in Site Seer or Google Scholar and that sort of research search place.

If you are looking for more recent stuff Jeff Dean is a Google fellow and he does amazing talks. He will often talk about amazing things like every programmer should know the difference between how long it takes to do a disc seek versus looking something up in RAM versus doing something across data centers across the country. He does these really great talks so I would look up anything by Jeff Dean.

Another person to do some research on is Oris Osley. He often writes about our data center architecture. Luiz Barroso is another person. We’ll try to insert a few links inside the description of the video but also check out the Google Research team because when they are papers published by Googlers they will often add links to those papers.

A lot of people think it’s hard to find out things about how Google works. Not only do we blog, show up at conferences, listen on forums, do webmaster videos, all these things we also try to talk about what we do in the academic literature. We publish in IEEE and ACM, lots of good stuff along those lines. And so check into that a little bit because there really is a lot of good data about how Google operates in terms of the technology, the machinery and all that sort of stuff. Do check into it because it can be really interesting.

The video cannot be shown at the moment. Please try again later.

If we tried to fetch them all, and our architecture can almost support that then the web might crash from all of those requests. We try to crawl in a relatively polite way. We also try to prioritize based on things like the PageRank of a particular page or maybe a site might have a lot of PageRank.

The question is essentially if A is getting crawled a lot but the original article starts on B what if A rips off B? There are ways you can help to guard against that. For example if you do a Tweet people will see it. People may link to it and we may follow those links faster than we will discover it the other way.

Another thing you can do is hook up things like Pub Sub Hubub, which will ping various places. There is a very limited amount in which we will use Pub Sub Hubub to help improve our crawl and that might change over time. That’s a great way to asynchronously say hey there is a new article or there is a new blog post.

But let’s go ahead and play with this hypothetical scenario. If A has copied your article and changed the time stamp that’s a little bit deceptive you know. It’s as if they are claiming that they have written it so you can do a couple things.

If you have already authored that article you can always do what is known as a digital millennium copyright act, sort of notice, where you send in this DMCA request. You can find the information at Google.com/DMCA.html and basically what you are saying is this site copied me but I’m the original author. Now this other site can counter notify and say, “No I wrote this page,” which has some penalties to it if they are lying or they cannot dispute it and the stuff disappears off the other site. So if someone is ripping you off you can always to a DMCA notice.
You can also for example if it’s an auto generated site that is ripping off or scraping a bunch of people you can also do a SPAM report because that’s not a high quality site and it’s not the sort of thing we want to have in our index.

But, let’s just play it all the way out to the corner case, it is possible that we will find an article on one site before we find it on the other site. And so it is definitely the case that we try to hard to find out who the original creator of that particular piece of content but I wouldn’t claim that we are perfect. We do as much as I can think of to try to figure out what are the ways that people can indicate that they wrote the content.

In fact in Google News we just introduced a couple new tags, almost as an experiment to see how well it works to sort of say, here is the original author of this content. So there are approaches we are exploring to sort of figure out if there are other ways to do that but at least for the time being in theory it is possible to have an article. In practice it tends to not happen that often and you do have ways you can get around that or ways that you can take action from a DMCA request all the way up to a SPAM report.

Hope that helps.

The video cannot be shown at the moment. Please try again later.