robots.txt – Why is Google Showing the Pages I Have Blocked?

Posted • October 17, 2009 • 1 Comment

First time visiting Search Engine Optimizician? You might want to subscribe to the RSS feed for updates on this topic.

Video from Google’s Matt Cutts explaining why your blocked pages are showing up in search results.

Matt Cutts A robots.txt file restricts access to your site by search engine bots that crawl the web. While Google won’t crawl or index the content of pages blocked by robots.txt, they may still index the URLs if they are found in other pages on the web.

To use a robots.txt file, you’ll need to have root access to your server.

A robot.txt file can be very useful. It is used to tell the search engines what folders to stay out of and where your sitemap.xml file is located. If you have an area of your site you don’t want displayed by the search engines this is what you use to accomplish it. ( e.g. you have special landing pages for your pay-per-click or other campaigns that contain duplicate content search engines might penalize you for duplicate content.)

The correct syntax for the robots.txt file is:
# /robots.txt file for http://www.yourdomain.com/
(Although not necessary, I use this statement commented out, because all the files must be named the same and I like to know which site it is that I am working on.)

User-agent: *
Disallow: /awstats/
Disallow: /cgi-bin/
Disallow: /Scripts/
Disallow: /formmail/
Disallow: /problem/
Disallow: /webalizer/
Disallow: /wordpress/
Disallow: /contact/
Sitemap: http://www.yourdomain.com/sitemap.xml

For more information on the use of robots.txt and other important files visit the Top Ten Rules for Beginning SEO
(Post from Search Engine Optimizician.)
Google will show robots.txt blocked URLs even if they haven’t crawled them.
Google just won’t show any summary from the page. They will however show related information from sites like DMOZ.

If really don’t want the page to show up in search results use the no index meta tag
<META NAME=”ROBOTS” CONTENT=”NOINDEX,NOFOLLOW”>
or use the URL removal tool inside of Google Webmaster Tools.

The meta robots tag will actually remove previously indexed URLs. Google’s information on removing URLs is here.

This answers a couple questions such as:
- Why is my url showing up in Google when I blocked it in robots.txt? Did you fetch that url?
- How do I make that url disappear from Google?

For even more information about blocking or removing pages using a robots.txt file visit Google Webmaster Central.

Late,
Gary Pool

Copy and paste this code into your website to create a link to this page.

Terms related to this post:

cache:Xk9PZr-bOvcJ:www searchengineoptimizationportland com/blog/2009/10/email-scam-uk-imf/ 09/10Email Address
cache:Xk9PZr-bOvcJ:www searchengineoptimizationportland com/blog/2009/10/email-scam-uk-imf/ address of imf office in uk
cache:Xk9PZr-bOvcJ:www searchengineoptimizationportland com/blog/2009/10/email-scam-uk-imf/ john @yahoo com@gmail com working in company or office in 2010
cache:Xk9PZr-bOvcJ:www searchengineoptimizationportland com/blog/2009/10/email-scam-uk-imf/ john email online in 2010@yahoo com@hotmail com
cache:Xk9PZr-bOvcJ:www searchengineoptimizationportland com/blog/2009/10/email-scam-uk-imf/ name of companies owners in cities of ukfaxemail@yahoo com @hotmail com

Don't keep it to your self share this post with others

Comments

One Response to “robots.txt – Why is Google Showing the Pages I Have Blocked?”

Tweets that mention Google : robots.txt : Search Results : Search Engine Optimizician -- Topsy.com
October 17th, 2009 @ 2:23 pm

[...] This post was mentioned on Twitter by Gary Pool and Alltop, Aggie Greiter. Aggie Greiter said: robots.txt – Why is Google Showing the Pages I Have Blocked? http://bit.ly/GKWPw [...]

@garypool

Heading to Dutch Bros on MLK in Portland for some great coffee. about 1 day ago from Ping.fm
Heading out to the regular mo. BizNet noon meeting at the South Shore Clubhouse, 12221 No. Westshore Drive. Behind the Jantzen Beach Mall. about 1 day ago from Ping.fm
Bing and Yahoo! Conjugate http://ping.fm/4KlD9 about 1 day ago from Ping.fm
Having a wonderful birthday lunch with my beautiful wife at Jakes Crawfish, downtown Portland. about 2 days ago from Ping.fm
Finally back at Best Baguette on Powell. Great sandwiches. Yummm*** about 2 days ago from Ping.fm
As Bing and Yahoo! get ready to climb in bed together what effect this will have on SEO. http://www.seopdx.com/blog/?p=5279 about 3 days ago from API
Men In Black is still a really good movie. It doesnt matter how old it is. about 3 days ago from Ping.fm
From @Sphinn: Exorcist Found to Exorcise Seth Godin from This Blog http://sphinn.com/story/157787 about 4 days ago from Power Twitter
Chargers! That was a real whuppin! Saints look good. Chargers not so good! :-( about 5 days ago from TweetDeck
Same goes for a football game :-( @CHRISVOSS about 5 days ago from TweetDeck
You had better believe it. RT @CHRISVOSS: There are no more hours in a bad day than in a good one! Unknown about 5 days ago from TweetDeck