Major search engines like Google and Yahoo provide webmaster tools that assist you in evaluating your links and content on an ongoing basis. Smart webmasters use these tools regularly to ensure their site is in top condition to optimize the experience for both visitors and search engine crawlers. In this post, I'll take a look at the Google Webmaster Tools and show you how to use them to look at search queries and crawler errors and what those mean to you.

The first step is to sign up at http://www.google.com/webmasters. Once your are logged in, the next step is to add your site by clicking the "Add a site..." graphic. We're going to use my personal site as an example.

After clicking Continue, you will be asked to verify your site to provide that you are indeed the owner. You can do this by placing the given meta tag in the header of your home page or by creating a new HTML file in the webroot of your site with a unique name provided by Google.

Once you choose the method that works best for you, click Verify. If the verification process is successful, you'll be taken to your dashboard where a snapshot of the top search queries, crawl errors, and the breakdown of the incoming links to your site are displayed. There is also a link to "Submit a Sitemap", but we'll get to that in a minute.

Next to each search query phrase is a number. That's your position in the Google search results. Obviously, the lower the number the better! You can use this table to see where certain keyword phrases are benefiting you and where others are not. If you write a lot about a certain keyword or topic then you'd like to see that phrase be present with a low number next to it. If that isn't the case, then you've got some search engine optimization work to do. Clicking "more" will show you a complete list of search queries for which your site was shown in Google's search results. Here are my first 17:

OK, click back to the dashboard. Search engine crawler errors are always very concerning to me. How are people going to find my content if the search engines can't, and how bad am I going to be penalized by the search engine because they think my site is unprofessional with bad content and links? Those are scary questions indeed if your company depends on the quality of your site for revenue.

And speak of the devil, I see there are 4 "Not found" errors reported by Google for my site! Let's find out why and fix it! Clicking "Not found" takes me to a screen detailing the URLs that generated the errors and the number of pages where links to them were found. Clicking the number of pages gives me an actual list of all the pages. The first bad link in my list is http://www.treyconnell.com/2007/content/view/20/29/. I see that it's linked to from this page: http://www.treyconnell.com/2007/02. Sure enough, when I view that page, I see an old link from when that content was on an old site. I never bothered to fix it when I ported the content to my new site. So what action do I take? There are 2 critical steps:

1. Fix the link or remove it if it should no longer go anywhere (duh).
2. Add a permanent (301) redirect to your web server that redirects all requests to the old, bad link to the new, updated link.

So why the 301 redirect? Google will recrawl from time to time, and it will see that you've permanently fixed the link and that it now points to valid content. You've effectively informed Google that you are actively maintaining your site and fixing errors. This is a good thing! NOTE: Do NOT use a temporary (302) redirect as search engines do not give that type of redirect any credit and will not give you any either.

So now you have the basics for setting up Google Webmaster Tools and effectively using them to analyze search queries and crawler errors. In my next post, I'll dive deeper into the additional features within the tools including sitemaps, crawler access and the robots.txt file, and internal linking.