SEOPDX • SEO Services • SEM Services • email Marketing • SEOSD • SEO Web Design
Free Reports • Free Video SEO Site Review • Subscribe to RSS • email SEOPDX • Sitemap

robots.txt – Why is Google Showing the Pages I Have Blocked?

Posted • October 17, 2009 • 1 Comment

Video from Google’s Matt Cutts explaining why your blocked pages are showing up in search results.

Matt Cutts A robots.txt file restricts access to your site by search engine bots that crawl the web. While Google won’t crawl or index the content of pages blocked by robots.txt, they may still index the URLs if they are found in other pages on the web.

To use a robots.txt file, you’ll need to have root access to your server.

A robot.txt file can be very useful. It is used to tell the search engines what folders to stay out of and where your sitemap.xml file is located. If you have an area of your site you don’t want displayed by the search engines this is what you use to accomplish it. ( e.g. you have special landing pages for your pay-per-click or other campaigns that contain duplicate content search engines might penalize you for duplicate content.)

The correct syntax for the robots.txt file is:
# /robots.txt file for http://www.yourdomain.com/
(Although not necessary, I use this statement commented out, because all the files must be named the same and I like to know which site it is that I am working on.)

User-agent: *
Disallow: /awstats/
Disallow: /cgi-bin/
Disallow: /Scripts/
Disallow: /formmail/
Disallow: /problem/
Disallow: /webalizer/
Disallow: /wordpress/
Disallow: /contact/
Sitemap: http://www.yourdomain.com/sitemap.xml

For more information on the use of robots.txt and other important files visit the Top Ten Rules for Beginning SEO
(Post from Search Engine Optimizician.)
Google will show robots.txt blocked URLs even if they haven’t crawled them.
Google just won’t show any summary from the page. They will however show related information from sites like DMOZ.

If really don’t want the page to show up in search results use the no index meta tag
<META NAME=”ROBOTS” CONTENT=”NOINDEX,NOFOLLOW”>
or use the URL removal tool inside of Google Webmaster Tools.

The meta robots tag will actually remove previously indexed URLs. Google’s information on removing URLs is here.

This answers a couple questions such as:
- Why is my url showing up in Google when I blocked it in robots.txt? Did you fetch that url?
- How do I make that url disappear from Google?

For even more information about blocking or removing pages using a robots.txt file visit Google Webmaster Central.

Late,
Gary Pool

Copy and paste this code into your website to create a link to this page.

Don't keep it to your self share this post with others

Comments

One Response to “robots.txt – Why is Google Showing the Pages I Have Blocked?”

Tweets that mention Google : robots.txt : Search Results : Search Engine Optimizician -- Topsy.com
October 17th, 2009 @ 2:23 pm

[...] This post was mentioned on Twitter by Gary Pool and Alltop, Aggie Greiter. Aggie Greiter said: robots.txt – Why is Google Showing the Pages I Have Blocked? http://bit.ly/GKWPw [...]

@garypool

As a SEO we must be aware of most if not all search engines our potential traffic might use. 8-) ( Please vi.. http://tinyurl.com/yc5ptu5 about 2 hours ago from Ping.fm
Back at http://crossroadscoffeecafe.com with the No-name Networking Group on Wednesdays again. ;-) about 3 hours ago from Mauku
As a SEO we must be aware of most if not all search engines our potential traffic might use. 8-) http://bit.ly/9zoZo7 about 4 hours ago from twitterfeed
You either have to be first to market or have the best product. This just might be both! 8-) ( Please visit .. http://tinyurl.com/yfxab24 about 1 day ago from Ping.fm
email scams and internet scams - verify urban legends and email scams. ( Please visit the Search Engine Opti.. http://tinyurl.com/yffvdgo about 2 days ago from Ping.fm
How influential is social media to customer service and what can you do about it. http://bit.ly/c7UdOM about 2 days ago from twitterfeed
Good Info - “Call to Action” Buttons: Guidelines, Best Practices and Examples - http://tinyurl.com/ybj38er 8-) about 3 days ago from Power Twitter
@FirstDigg I agree! I have a great espresso machine at home. Yummy quad black americanos are killer. ;-) about 3 days ago from Power Twitterin reply to FirstDigg