Video from Google’s Matt Cutts explaining why your blocked pages are showing up in search results.

A robots.txt file restricts access to your site by search engine bots that crawl the web. While Google won’t crawl or index the content of pages blocked by robots.txt, they may still index the URLs if they are found in other pages on the web.

To use a robots.txt file, you’ll need to have root access to your server.

A robot.txt file can be very useful. It is used to tell the search engines what folders to stay out of and where your sitemap.xml file is located. If you have an area of your site you don’t want displayed by the search engines this is what you use to accomplish it. ( e.g. you have special landing pages for your pay-per-click or other campaigns that contain duplicate content search engines might penalize you for duplicate content.)

The correct syntax for the robots.txt file is:
# /robots.txt file for http://www.yourdomain.com/
(Although not necessary, I use this statement commented out, because all the files must be named the same and I like to know which site it is that I am working on.)

User-agent: *
Disallow: /awstats/
Disallow: /cgi-bin/
Disallow: /Scripts/
Disallow: /formmail/
Disallow: /problem/
Disallow: /webalizer/
Disallow: /wordpress/
Disallow: /contact/
Sitemap: http://www.yourdomain.com/sitemap.xml

For more information on the use of robots.txt and other important files visit the Top Ten Rules for Beginning SEO
(Post from Search Engine Optimizician.)
Google will show robots.txt blocked URLs even if they haven’t crawled them.
Google just won’t show any summary from the page. They will however show related information from sites like DMOZ.

If really don’t want the page to show up in search results use the no index meta tag
<META NAME=”ROBOTS” CONTENT=”NOINDEX,NOFOLLOW”>
or use the URL removal tool inside of Google Webmaster Tools.

The meta robots tag will actually remove previously indexed URLs. Google’s information on removing URLs is here.

This answers a couple questions such as:
- Why is my url showing up in Google when I blocked it in robots.txt? Did you fetch that url?
- How do I make that url disappear from Google?

For even more information about blocking or removing pages using a robots.txt file visit Google Webmaster Central.

Late,
Gary Pool

robots.txt – Why is Google Showing the Pages I Have Blocked? is a post stolen from the Search Engine Optimizician

So you didn’t attend the Web 2.0 Summit in San Francisco?
Don’t fret this is valuable informative video will get you caught up and is well worth your investment of time. Matt’s video is also being touted as being of interest to all levels of SEOs.

Matt Cutts from Google posted this twenty minute video sharing with us the talk In November 2008 on a panel about “Preventing Virtual Blight” at the Web 2.0 Summit in San Francisco. During this video Matt goes over what has been coined to be “virtual blight,” or when a site or community becomes infected with spam. Matt also tells us how to take advantage of using the blackhat SEO mindset to keep our sites from being considered just another one of the blackhat websites.

Preventing Virtual Blight

• web spam
• parasitic SEO
• black hats and white hats
• don’t let people insert code on your site (javascript can crate a sneaky redirect)
• do some security reviews

Use the Black Hat SEO Mindset

• any successful system will bring out the spammers
• if you’re being attacked, you’re doing something right
• think like a black hat now to save yourself anguish

I’m making a resolution this year that when I do a substantial (not just Q&A) presentation at a conference, I’ll try to recreate a version of the talk later on for the people who couldn’t attend the conference. I’d like Google to communicate more and more this year, so this is another step to help with that.

You can also view the presentation slides Matt used or watch the slides below:

Solutions:

• patch religiously (e.g. WordPress, Joomla)
• ask webhost: any security break-ins? What cpanel version?
• set up an alert for [site:yourdomain.com viagra]
• check your server logs for weird requests
• check your cached page on Google
• check top queries at google.com/webmasters

Late,
Gary Pool

How to use the Black Hat SEO Mindset – Video By Matt Cutts from Google is a post stolen from the Search Engine Optimizician

I have been to so many conferences in the last ten years and I couldn’t even count them all.

Like Matt Cutts says in his recent blog post:

I don’t even know how many conferences I’ve been to in the last decade, but it’s probably 30-40. In that time, maybe 2-3 conferences have really nailed the conference badge for attendees. Here’s what the ideal conference badge should look like, in my opinion:

• Each attendee’s first name needs to be large and easily readable
• Make the badge big
• Put the attendee’s name on the front and the back of the badge

I bring my own badge and lanyard to all of these conferences and it solves the inferior badge problems. I usually have to attach their badge below the one I have produced. Everything is large enough to read. I have used this at conferences over the past three years and have had no objections.

I have two lanyards that I use, a green one form the Internet Strategy Forum and a blue one from Meetings Market Academy. They both have a place for your own business cards, business cards you collect and a spot for a pen. Like a boy scout I always come prepared.

Matt also sites Mike Davidson’s take on the right way to do a conference badge.

• Too small at 3 inches tall by 4 inches wide. Bump that puppy up to 4 by 6.
• First and last name on same line.
• Set in 25-ish point Arial: the worst font in the world and even more unreadable at that size and weight.
• Title of conference — the least important information on the badge — dominates the space.
• “ATTENDEE” is the boldest text on the badge. It’s not *that* important as speakers and attendees share 99% of the same privileges.
• Lanyard contains name of conference instead of being sold to a sponsor. No one cares what the lanyard looks like, so go ahead and sell it to Yahoo or something.

Mike even offers an illustrator file in eps format to download and be used to help bring people closer together at your conference.

Late,
Gary Pool

Conference Badges, Matt Cutts from Google is a post stolen from the Search Engine Optimizician

SES San Jose to follow or to nofollow that is the question.

Paid links were the most controversial topic at this years SES San Jose.

Here is a great, although 18 minute long video on Google’s paid link policy. The video is by WebProNews MC’d by Michael McDonald. The video features SEOmoz.com’s Rand Fishkin and Wolf-Howl.com’s Michael Gray. The topics: paid links and the effects of Google using paid links.

In a post by Rand, he includes an “if I were Matt Cutts” quote.

Paid links, in Google’s opinion, pollute the search results. The people on the search quality team are spending a lot of time and brainpower fighting back by detecting and devaluing paid links. Thus, if you’re a webmaster, we’d urge you not to buy links for the perceived value that you think they might provide in helping to rank your site better. We’ve got a ton of very smart people cutting off the flow of link juice from those links and you’re going to be wasting your money. If you want to buy links for traffic or branding or other marketing purposes, that’s great, but be aware that we’re not going to count them in our rankings.

In my opinion Google is just jones-ing to corner the market on the value of paid links. They sell paid links how dare anyone else be able to take advantage of this without them being able to take their slice of pie. But, this is the way you would expect a monopoly to react anyway, isn’t it?

late,
gary pool

Google | nofollow | SES is a post stolen from the Search Engine Optimizician

While doing my homework this morning I ran across these videos. They were posted by John Pozadzides on his One Man’s Blog. There are some great SEO tidbits in these videos. Matt gives us some great insight.

This is a video of the one hour long lecture given by Matt Cutts at the WordCamp 2007 conference in San Francisco, CA two weeks ago on the topic of Whitehat SEO tips for Bloggers. For those not familiar, SEO stands for Search Engine Optimization and this lecture is all about ways to make your Web site rank higher in Google and the other search engines.

The videos were filmed at WordCamp 2007 conference in San Francisco, CA. WordCamp was a 2-day conference for WordPress users and developers. The first day’s focus was on how to be a better blogger, with the second day was focused on the development and future of WordPress.

If you watch closely these videos totaling 1 hour Matt will give you some great insight into the world of Google SEO.

Whitehat SEO tips for Bloggers – Part 1

Whitehat SEO tips for Bloggers – Part 2

So how great was that search engine optimization information? Did you see how the blogging SEO information you picked up here will translate into the SEO for the rest of your sites?

You can actually download Matt’s PowerPoint presentation here.

Thanks John!

Late,
Gary Pool

Whitehat SEO tips for Bloggers | Matt Cutts | WordPress is a post stolen from the Search Engine Optimizician

Search Engine Marketing Professionals of Portland is an organization based in Portland, Oregon whose mission is to inform and educate area businesses on the benefits of SEM.

I attended the SEMpdx Networking Event at Paddy’s Bar & Grill (65 SW Yamhill St., Portland, OR 97204). The Topic I heard the most about was our annual event SearchFest. Last years SearchFest 2007 was an incredible event. Next Years promises to be even better.

SearchFest 2008 was a hot topic at this networking event. Some of the names I over heard being thrown around were Matt Cutts, Rand Fishkin, Aaron Wall, Chris Sherman, Cindy Krum, Catherine Seda, John Battelle, Heather Lloyd-Martin, Vanessa Fox, Guy Kawasaki, Scott Hendison, and Danny Sulivan.

I can’t wait for the actual lineup to be released. SearchFest 2008 will be hosted at the fantastic facilities at the Oregon Zoo. It is going to be a day long catered event. The facilities are plush and even have an open air covered patio right off the main lobby. SearchFest 2008 is scheduled to be held March 10, 2008. If you want to attend this marvelous event I would suggest you get your tickets early.

The next SEMpdx event will be the SEM Hot Seat. It will be heald Tuesday, August 14 2007, Registration & Networking: 6:00 PM, Program: 6:30 – 8:00 PM. This Hot Seat will be held at the Hotel DeLuxe (729 SW 15th Ave. Portland, OR 97205). Get real search marketing advice from experienced SEO & PPC experts. Take part in SEM Hot Seat, an intimate panel event where you’ll get real, specific search engine marketing advice tailored to your site at a bargain price. You can also just attend to hear the experts go through their paces.

I’ll see you at the SEMpdx Hot Seat

Late,
Gary Pool

SearchFest 2008 is a post stolen from the Search Engine Optimizician

I was surfing the net and watching some older Matt Cutts videos looking for some tidbits. They are all very informative and well worth watching. Here is a list with links.

• Crawl dates in Google cache
  4 min – Sep 6, 2006
  Matt talks about how Googlebot crawls the web, and what "crawl date" is shown on cached pages. In this video: – Red candy is a 404 page – Purple candy is a 200 (OK) page – Green candy is a 304 status code (page has not changed)
• Qualities of a good site
  6 min – Jul 30, 2006
  Matt Cutts answers Google questions: – Does Sitemaps depend on pageviews? – What are the top things to do in SEO? – How does Google decide to use a DMOZ snippet versus the description from a meta tag? – Should I use bold or strong tags?
• Optimize for Search Engines or Users?
  4 min – Jul 30, 2006
  Matt discusses SEO VS. User Opitmization, Site Spam, and Code Validation and the W3C.
• Recap of SES San Jose 2006
  9 min – Aug 28, 2006
  A short review of news from SES San Jose, and some educational or funny moments.
  Some of the changes happening to people or companies recently – Fun moments
• Matt Cutts Warns Against Buying Links
  51 sec – Jun 15, 2007
  Matt explains that link buying IS against the practice of link buying and warns those who are participating in such schemes may face tough times ahead as Google works harder to remove them.

During my Google searching of various SEO topics I ran across a series of videos with the keyword "Algorithm" using the related topics link. Curiosity got the best of me, so I watched some of the videos on youtube. These are great. I even uploaded one of the videos to myspace. It’s called the Algorithm March check it out.

This is a clip from a japanese kids show called "Pythagoras Switch" or "Pitagora Suicchi".

One last tidbit. Searching this morning I found a Google result for a Matt Cutts blog post that wasn’t uploaded yet… Uhmm.

Matt Cutts: Gadgets, Google, and SEO
I also got Matt-jacked into a dark room by Rand for the “SMX Diaries” video interviews. And WebProNews had some snippets of me from the Q&A session. …
www.mattcutts.com/blog/ – Jun 19, 2007

The latest post on his blog is or was:

Terry Semel stepping down
June 18, 2007 @ 1:45 pm · Filed under Web/Net

Well thats the story of Matt Cutts MySpace, SEO, Youtube, and the Algorithm March.

Late,
Gary Pool

The Story of Matt Cutts, MySpace, SEO, Youtube, and the Algorithm March is a post stolen from the Search Engine Optimizician

Everybody’s talking about “Supplemental Hell” now days, with Forbes reporter Andy Greenberg’s very biased article “Condemned To Google Hell” this week, Rand is jumping on the bandwagon with his Whiteboard Friday this week. Rand also gives us some secret information on the topic shh!

Whiteboard Friday – “Supplementary My Dear Watson” from SEOmoz and Rand Fishkin.

Some thoughts from Matt Cuts:

I’d agree with most of your video and your bullet points, but lean more toward Michael’s viewpoint on bullet point #2; duplicate content doesn’t make you more likely to have pages in the supplemental index in my experience. It could be a symptom but not a cause, e.g. lots of duplicate content implies lots of pages, and potentially less PageRank for each of those pages. So trying to surface an entire large catalog of pages would mean less PageRank for each page, which could lead to those pages being less likely to be included in our main web index. I’m not aware of an explicit mechanism whereby duplicate content is more likely to be in our supplemental results, but I’m also happy to admit that as supplemental results are different from webspam, I’m not the expert at Google on every aspect of supplemental results.

One fine point to make: when people see the “show duplicate results” link at the end of search results and click it to add “&filter=0″ as an extra parameter, the new results you see are not all from the supplemental results. Lemme see if I can find a query to demonstrate that. Ah, here we are, the first one I tried. [site:mattcutts.com foxmarks] returns one result. If I click to see more results, that post has also been indexed at other urls, but at least two of the extra urls are in our main web index, not the supplemental index.

Maybe it’s not really Google Hell after all, just purgatory?

late,
Gary

Google’s Supplemental Hell Exposed? is a post stolen from the Search Engine Optimizician